M-estimator

Definition

Consider an error (loss) function $\rho (\cdot )$, The M-estimator is obtained by ${\hat{\mathbf{\beta }}}_M=\underset{\mathbf{\beta }}{\mathrm{argmin}}\sum _{i=1}^n\rho \left(\frac{{ϵ}_i}{s}\right)=\underset{\mathbf{\beta }}{\mathrm{argmin}}\sum _{i=1}^n\rho \left(\frac{y_i-{\mathbf{x}}_i^{⊺}\mathbf{\beta }}{s}\right)$ where $s$ is a robust estimator, and given by $s=\frac{\mathrm{med}(|e_i-\mathrm{med}(e_i)|)}{0.6745}$

If $\rho (u)=\frac{1}{2}{u}^2$, then it is LSE, and if $\rho (u)=|u|$, then it is $L_1$-norm regression estimator. The derivative of $\rho$, denoted $\psi =\frac{\partial \rho }{\partial \mathbf{\beta }}$, is called the influence function. Since $\psi$ is a non-liner in $\beta$, we van not get explicit solution. Instead, we use an iterative method, called IRLS (Iterative Reweighted Least Squares).

Iterative Reweighted Least Squares

1. Initialize ${\mathbf{\beta }}_0$, often obtained from Ordinary Least Squares. ${\hat{\mathbf{\beta }}}_0=({\mathbf{X}}^{⊺}\mathbf{X}{)}^{-1}{\mathbf{X}}^{⊺}\mathbf{y}$
2. Calculate the weight $w_{i,j}=\frac{\psi [(y_i-{\mathbf{x}}_i^{⊺}{\hat{\mathbf{\beta }}}_j)/s]}{(y_i-{\mathbf{x}}_i^{⊺}{\hat{\mathbf{\beta }}}_j)/s}$ If $y_i-{\mathbf{x}}_i^{⊺}\mathbf{\beta }=0$, then $w_i=1$
3. Update $\mathbf{\beta }$ using Weighted Least Squares ${\hat{\mathbf{\beta }}}_{j+1}=({\mathbf{X}}^{⊺}{\mathbf{W}}_j\mathbf{X}{)}^{-1}{\mathbf{X}}^{⊺}{\mathbf{W}}_j\mathbf{y}$ where ${\mathbf{W}}_j=\mathrm{diag}(w_{1,j},w_{2,j},\dots ,w_{n,j})$
4. Iterate step 1 to 3 until the estimator converges.