Least Squares Estimator

Definition

Simple Linear Regression Case

Consider a Simple Linear Regression model $Y_i={\beta }_0+{\beta }_1{X}_i+{ϵ}_i$ The least square estimator is the estimator that minimizes $\sum _{i=1}^n{ϵ}_i^2=\sum _{i=1}^n(Y_i-{\beta }_0+{\beta }_1{X}_i{)}^2$

The least square estimator of the model is ${\hat{\beta }}_0=\bar{Y}-{\hat{\beta }}_1\bar{X},{\hat{\beta }}_1=\frac{S_{XY}}{S_{XX}}$ where $S_{XY}=\sum _{i=1}^n(X_i-\bar{X})(Y_i-\bar{Y})$ and $S_{XX}=\sum _{i=1}^n(X_i-\bar{X}{)}^2$

Estimation of ${\sigma }^2$

$s^2=\frac{1}{n-2}{\sum }_{i=1}^n{e}_i^2$ where $e_i=(Y_i-{\hat{Y}}_i)$‘s are residuals

Multiple Linear Regression Case

Consider a Multiple Linear Regression model $\mathbf{y}=\mathbf{X}\mathbf{\beta }+\mathbf{ϵ}$ The least square estimator is the estimator that minimizes ${ϵ}^{⊺}ϵ=(\mathbf{y}-\mathbf{X}\mathbf{\beta }{)}^{⊺}(\mathbf{y}-\mathbf{X}\mathbf{\beta })$

The least square estimator of the model is $\hat{\mathbf{\beta }}=({\mathbf{X}}^{⊺}\mathbf{X}{)}^{-1}{\mathbf{X}}^{⊺}\mathbf{y}$

Fitted response vector is expressed as $\hat{\mathbf{y}}=\mathbf{X}\hat{\mathbf{\beta }}=\mathbf{X}({\mathbf{X}}^{⊺}\mathbf{X}{)}^{-1}{\mathbf{X}}^{⊺}\mathbf{y}$ where $H:=\mathbf{X}({\mathbf{X}}^{⊺}\mathbf{X}{)}^{-1}{\mathbf{X}}^{⊺}$ is called the hat matrix.

Estimation of ${\sigma }^2$

$s^2=\frac{{\mathbf{e}}^{⊺}\mathbf{e}}{n-p}$ where $\mathbf{e}=(e_1,e_2,\dots ,e_n)$, $e_i=(Y_i-{\hat{Y}}_i)$‘s are residuals, and $p$ is the number of the explanatory variables

Facts

The least square estimator ${\hat{\beta }}_1$ is a linear combination of $Y_i$‘s Let $w_i=\frac{(X-\bar{X})}{S_{XX}}$, then ${\hat{\beta }}_1=\sum _{i=1}^n{w}_i{Y}_i$

$E({\hat{\beta }}_1)={\beta }_1$ $E({\hat{\beta }}_0)={\beta }_0$ $\mathrm{Var}({\hat{\beta }}_1)=\frac{{\sigma }^2}{S_{XX}}$ $\mathrm{Var}({\hat{\beta }}_0)={\sigma }^2(\frac{1}{n}+\frac{{\bar{X}}^2}{S_{XX}})$

$\sum _{i=1}^n{e}_i=0$ $\sum _{i=1}^n{X}_i{e}_i=0,{\mathbf{X}}^{⊺}\mathbf{ϵ}=0$ $\sum _{i=1}^n{\hat{Y}}_i{e}_i=0,{\hat{\mathbf{y}}}^{⊺}ϵ=0$ The fitted line always go through $(\bar{X},\bar{Y})$

$E(\hat{\mathbf{\beta }})=\mathbf{\beta }$ $\mathrm{Cov}(\hat{\beta })=({\mathbf{X}}^{⊺}\mathbf{X}{)}^{-1}{\sigma }^2$ $E(s^2)={\sigma }^2$

Let $\mathbf{y}=(Y_1,\dots ,Y_n{)}^{⊺}$, where $Y_i$‘s are independent, $E(\mathbf{Y})=\mathbf{X}\mathbf{\beta }$, and ${\sigma }^2,{\mu }_3,{\mu }_4$ be the 2nd, 3rd, and 4th Central Moment of $Y_i$ respectively. Then, $s^2$ is the unique non-negative quadratic Unbiased Estimator of ${\sigma }^2$ with minimum variance when the excess kurtosis is $({\mu }_4-3{\sigma }^4)/{\sigma }^4=0$ or when the diagonal elements of the hat matrix $X(X^{⊺}X{)}^{-1}{X}^{⊺}$ are equal.