Gauss-Newton Method

Definition

Gauss-Newton method is an iterative algorithm used to solve used to solve non-linear least squares problems. This method approximates Hessian Matrix using the Jacobian Matrix.

Algorithm

Consider a non-linear regression model with $k$-dimensional explanatory variable and $p$-dimensional parameter vector $Y_i=f({\mathbf{X}}_i,\mathbf{\theta })+{ϵ}_i,i=1,2,\dots ,n$ The first order Taylor expansion gives the linear approximation of the model. $\mathbf{y}=\mathbf{\eta }({\mathbf{\theta }}_0)+{\mathbf{J}}_0(\mathbf{\theta }-{\mathbf{\theta }}_0)+ϵ$ where ${\mathbf{\theta }}_0=({\theta }_{1,0},{\theta }_{2,0},\dots ,{\theta }_{p,0}{)}^{⊺}$ is the initial vector for $\mathbf{\theta }$ given by domain knowledge, $\mathbf{\eta }({\mathbf{\theta }}_0)=[f({\mathbf{X}}_1,{\mathbf{\theta }}_0),f({\mathbf{X}}_2,{\mathbf{\theta }}_0),\dots ,f({\mathbf{X}}_n,{\mathbf{\theta }}_0){]}^{⊺}$ is the estimation vector with ${\mathbf{\theta }}_0$, and ${\mathbf{J}}_0=\left(\frac{\partial f({\mathbf{X}}_i,\mathbf{\theta })}{\partial {\theta }_j}\right){|}_{\mathbf{\theta }={\mathbf{\theta }}_0}$ is the $n\times p$ Jacobian Matrix of the function $f$ at $\mathbf{\theta }={\mathbf{\theta }}_0$.

Now, the approximated model has a Multiple Linear Regression. Since $(\mathbf{y}-\mathbf{\eta }({\mathbf{\theta }}_0))$ and ${\mathbf{J}}_0$ are constants given ${\mathbf{\theta }}_0$, we can find LSE of $(\mathbf{\theta }-{\mathbf{\theta }}_0)$. $({\hat{\mathbf{\theta }}}_1-{\hat{\mathbf{\theta }}}_0)=({\mathbf{J}}_0^{⊺}{\mathbf{J}}_0{)}^{-1}{\mathbf{J}}_0^{⊺}(\mathbf{y}-\mathbf{\eta }({\theta }_0))$ And the updating formula is defined as ${\hat{\mathbf{\theta }}}_{n+1}={\hat{\mathbf{\theta }}}_n-({\mathbf{J}}_n^{⊺}{\mathbf{J}}_n{)}^{-1}{\mathbf{J}}_n^{⊺}(\mathbf{y}-\mathbf{\eta }({\hat{\mathbf{\theta }}}_n))$ The updating is repeated iteratively until it converges. ${\hat{\mathbf{\theta }}}_i$ converges to the $\mathbf{\theta }$ minimizing Sum of Squared Errors Loss with a proper initial value ${\mathbf{\theta }}_0$.