Principal Component Analysis

Definition

PCA is a linear dimensionality reduction technique. The correlated variables are linearly transformed onto a new coordinate system such that the directions capturing the largest variance in the data.

Population Version

Given a random vector $\mathbf{x}$, we find a $\mathbf{\alpha }$ such that $\mathrm{Var}({\mathbf{\alpha }}^{⊺}\mathbf{x})$ is maximized: $\underset{\mathbf{\alpha }}{\mathrm{argmax}}\mathrm{Var}({\mathbf{\alpha }}^{⊺}\mathbf{x})\text{s.t.}{\mathbf{\alpha }}^{⊺}\mathbf{\alpha }=1$ Equivalently, by the Method of Lagrange Multipliers with ${\mathbf{\alpha }}^{⊺}\mathbf{\alpha }=1$, $\underset{\mathbf{\alpha }}{\mathrm{argmax}}{\mathbf{\alpha }}^{⊺}\Sigma \mathbf{\alpha }-\lambda ({\mathbf{\alpha }}^{⊺}\mathbf{\alpha }-1)$ By differentiation, the $\mathbf{\alpha }$ is given by the eigen value problem $\Sigma \mathbf{\alpha }=\lambda \mathbf{\alpha }$ Thus the $\mathbf{\alpha }$ maximizing the variance of ${\mathbf{\alpha }}^{⊺}\mathbf{x}$ is the eigenvector corresponding to the largest Eigenvalue.

Sample Version

Given a data matrix $\mathbf{X}$, by Singular Value Decomposition, A matrix $\mathbf{X}$ can be factorized as $\mathbf{X}=\mathbf{UD}{\mathbf{V}}^{⊺}$. By algebra, $\mathbf{XV}=\mathbf{UD}=:\mathbf{Z}\Rightarrow \mathbf{X}{\mathbf{v}}_i=d_i{\mathbf{u}}_i=:{\mathbf{z}}_i$, where we call ${\mathbf{z}}_i$ the $i$-th principal component.

Facts

Since $\mathrm{Var}({\mathbf{z}}_i)=\mathrm{Var}(\mathbf{X}{\mathbf{v}}_i)=\frac{d_i^2}{n}$ and $d_1\ge d_2\ge \cdots \ge d_p\ge 0$ $\mathrm{Var}({\mathbf{z}}_1)\ge \mathrm{Var}({\mathbf{z}}_2)\ge \cdots \ge \mathrm{Var}({\mathbf{z}}_p)\ge 0$