Ridge Regression

Definition

$\begin{array}{rl}{\hat{\beta }}_{\text{ridge}}& =({\mathbf{X}}^{⊺}\mathbf{X}+\lambda \mathbf{I}{)}^{-1}{\mathbf{X}}^{⊺}\mathbf{y}\\ & =\underset{\beta }{\mathrm{argmin}}(\mathbf{y}-\mathbf{X}\beta {)}^{⊺}(\mathbf{y}-\mathbf{X}\beta )+\lambda {\beta }^{⊺}\beta \\ & =\underset{\beta }{\mathrm{argmin}}(\mathbf{y}-\mathbf{X}\beta {)}^{⊺}(\mathbf{y}-\mathbf{X}\beta ),\text{subject to}{\beta }^{⊺}\beta \le c\end{array}$ where $\lambda \ge 0$ is a complexity parameter that controls the amount of shrinkage.

Ridge regression is particularly useful to mitigate the problem of Multicollinearity in linear regression

Facts

${\hat{\mathbf{y}}}_{\text{ridge}}=\mathbf{X}{\hat{\beta }}_{\text{ridge}}=\mathbf{X}({\mathbf{X}}^{⊺}\mathbf{X}+\lambda \mathbf{I}{)}^{-1}{\mathbf{X}}^{⊺}\mathbf{y}=\sum _{i=1}^p{\mathbf{u}}_j\frac{d_j^2}{d_j^2+\lambda }{\mathbf{u}}_j^{⊺}\mathbf{y}$ where $\mathbf{X}=\mathbf{UD}{\mathbf{V}}^{⊺}$ by Singular Value Decomposition