Bayes Estimator

Definition

${\hat{\delta }}_{\text{Bayes}}=\underset{\delta }{\mathrm{argmin}}r(\theta ,\delta )=\underset{\delta }{\mathrm{argmin}}\rho (\pi ,\delta )$

Estimator that minimizes the Bayes Risk $r(\theta ,\delta )$ or Posterior Risk $\rho (\pi ,\delta )$

Facts

Under Squared Error Loss, the Bayes estimator is a posterior mean, and a posterior mode under Absolute Error Loss.

Consider a regression model $\mathbf{y}\sim N_n(\mathbf{X}\mathbf{\beta },{\sigma }^2\mathbf{I})$ and the prior distribution for $\mathbf{\beta }$, $\mathbf{\beta }\sim N_p(\mathbf{m},{\sigma }^2\mathbf{V})$. Then, the Bayes estimator under the Squared Error Loss is obtained as ${\hat{\mathbf{\beta }}}_{\text{Bayes}}=({\mathbf{X}}^{⊺}\mathbf{X}+{\mathbf{V}}^{-1}{)}^{-1}({\mathbf{V}}^{-1}\mathbf{m}+{\mathbf{X}}^{⊺}\mathbf{y})$ If $\mathbf{V}={\lambda }^{-1}{\mathbf{I}}_p$ for some $\lambda >0$ and $\mathbf{m}=0$, then the Bayes estimator is the same as ridge estimator ${\hat{\mathbf{\beta }}}_{\text{Bayes}}={\hat{\mathbf{\beta }}}_{\text{ridge}}=({\mathbf{X}}^{⊺}\mathbf{X}+\lambda \mathbf{I}{)}^{-1}{\mathbf{X}}^{⊺}\mathbf{y}$ If $\mathbf{V}={\lambda }^{-1}({\mathbf{X}}^{⊺}\mathbf{X}{)}^{-1}$ and $\mathbf{m}=0$, then the Bayes estimator is the James-Stein regression estimator ${\hat{\mathbf{\beta }}}_{\text{Bayes}}=(1+\lambda {)}^{-1}\hat{\mathbf{\beta }}$