Kernel Principal Component Analysis

Definition

Consider an $n\times p$ demeaned matrix $\mathbf{X}$ By SVD $\mathbf{X}=\mathbf{UD}{\mathbf{V}}^{⊺}$, where $\mathbf{U},\mathbf{V}$ are orthonormal matrices and $\mathbf{D}$ is a Diagonal Matrix. Then, $\mathbf{X}{\mathbf{X}}^{⊺}=\mathbf{UD}{\mathbf{V}}^{⊺}(\mathbf{UD}{\mathbf{V}}^{⊺}{)}^{⊺}=\mathbf{U}{\mathbf{D}}^2{\mathbf{U}}^{⊺}$ By PCA $\mathbf{Z}:=\mathbf{XV}=\mathbf{UD}$. For a linear kernel $\mathbf{K}=\mathbf{X}{\mathbf{X}}^{⊺}$, $\mathbf{K}=\mathbf{Z}D{\mathbf{U}}^{⊺}\Rightarrow K(D{\mathbf{U}}^{⊺}{)}^{-1}=\mathbf{Z}$. $\because \mathbf{U}$ is an Orthonormal Matrix, $\mathbf{Z}=\mathbf{KU}{\mathbf{D}}^{-1}\iff {\mathbf{z}}_{im}=\sum _{j=1}^n\frac{{\mathbf{u}}_{jm}}{d_m}K({\mathbf{x}}_i,{\mathbf{x}}_j)$ The kernel principal components are given by solution of the optimization problem. $\underset{g_i\in {\mathcal{H}}_K}{\max }\mathrm{Var}{g}_i(\mathbf{X})\text{subject to}||g_i|{|}_{{\mathcal{H}}_K}=1,\forall j<i,⟨g_i,g_j{⟩}_{{\mathcal{H}}_K}=0$ where $g_i(\mathbf{x})=\sum _{j=1}^n\frac{{\mathbf{u}}_{ji}}{d_i}K(\mathbf{x},{\mathbf{x}}_j)$