Local Polynomial Regression

Definition

Local polynomial regression fits the polynomial function with the data near the given point. The polynomial function has a form similar to Taylor Series to exploit the relation between the coefficients and the differentiation at the given point.

The $p$-th degree polynomial function at a point $x_0$ is defined as $f(x_0)=\sum _{i=0}^p{\beta }_i(x_0-x{)}^i={\beta }_0+{\beta }_1(x_0-x)+\cdots +{\beta }_p(x_0-x{)}^p$ where $p\in {\mathbb{N}}_0$

Local polynomial regression estimator is obtained by the relation $\hat{f}(x)={\hat{\beta }}_0$. The coefficients are estimated by an optimization problem $\hat{\mathbf{\beta }}=\underset{\mathbf{\beta }}{\mathrm{argmin}}\sum _{i=1}^n\left([Y_i-\sum _{j=0}^p{\beta }_i(X_i-x{)}^j{]}^{2}K\left(\frac{X_i-x}{\lambda }\right)\right)$ where $\mathbf{\beta }=({\beta }_0,\dots ,{\beta }_p)$, $K$ is a kernel function, and $\lambda$ is a smoothing parameter.

In a matrix notation, the optimization problem can be written as $\hat{\mathbf{\beta }}=\underset{\mathbf{\beta }}{\mathrm{argmin}}(\mathbf{y}-\mathbf{X}\mathbf{\beta }{)}^{⊺}\mathbf{W}(\mathbf{y}-\mathbf{X}\mathbf{\beta })$ where $\mathbf{y}=(Y_1,\dots ,Y_n{)}^{⊺}$, $\mathbf{X}=\left(\begin{array}{ccc}(X_1-x{)}^0& \dots & (X_1-x{)}^p\\ ⋮& \ddots & ⋮\\ (X_n-x{)}^0& \dots & (X_n-x{)}^p\end{array}\right)$ is an $n\times (p+1)$ matrix, and $\mathbf{W}=\mathrm{diag}\left[K\left(\frac{X_1-x}{\lambda }\right),\dots ,K\left(\frac{X_n-x}{\lambda }\right)\right]$

A solution of the problem is obtained as ${\hat{f}}^{(j)}(x)=j!{\hat{\beta }}_j=j!1_j^{⊺}({\mathbf{X}}^{⊺}\mathbf{WX}{)}^{-1}{\mathbf{X}}^{⊺}\mathbf{Wy}$ where $1_j$ is a $(p+1)$-vector whose $j$-th element is $1$ and the others are $0$.