Series Estimator

Definition

Series Estimator

Suppose a non-parametric regression function $y_i=f(x_i)+{ϵ}_i$ where $f(x)\in L_2[a,b]$ and let a sequence of functions $\{g_j(x)\}$ be a complete orthonormal basis. Then, the function can be written as a series. $f(x)=\sum _{j=1}^{\infty }{\theta }_j{g}_j(x)$ where ${\theta }_j=\int g_j(x)f(x)dx$ is a parameter subject to estimation.

The regression function can be approximated with the estimated parameter ${\hat{\theta }}_j$ and finite sum. It is called a series estimator ${\hat{f}}_{\lambda }(x)=\sum _{j=1}^{\lambda }{\hat{\theta }}_j{g}_j(x)$ where $\lambda \in \mathbb{N}$ is a smoothing parameter.

The series estimator can be written as a form of a kernel estimator with the estimation of the parameter ${\hat{\theta }}_j=\frac{1}{n}\sum _{i=1}^n{g}_j(x_i)Y_i$. ${\hat{f}}_{\lambda }(x)=\sum _{j=1}^{\lambda }{\hat{\theta }}_j{g}_j(x)=\frac{1}{n}\sum _{i=1}^n\left(\sum _{j=1}^{\lambda }{g}_j(x_i)g_j(x)\right)Y_i=\frac{1}{n}\sum _{i=1}^n{K}_{\lambda }(x,x_i)Y_i$ where $K_{\lambda }$ is a kernel.

Thresholding

In general, as the $j$ is larger the $|{\theta }_j|$ smaller. So the terms of the series can be selected with a certain algorithm.

The hard-thresholding estimator is defined as ${\hat{f}}_H(x)=\sum _{j=1}^{\infty }{\hat{\theta }}_j{I}_{\{|{\hat{\theta }}_j|>\delta \}}\cdot g_j(x)$ where $\delta$ is a thresholding parameter

And the soft-thresholding estimator is defined as ${\hat{f}}_S(x)=\sum _{j=1}^{\infty }\mathrm{sign}({\hat{\theta }}_j)(|{\hat{\theta }}_j|-\delta {)}_{+}\cdot g_j(x)$