Probabilistic PCA

Definition

Probabilistic PCA (PPCA) is a dimensionality reduction techniques that that aims to explain the covariance structure of given data using a fewer number of latent variables (features). PPCA is a probabilistic formulation of PCA. Instead of finding orthogonal directions that maximize variance, PPCA models the observed data as linear combinations of latent features plus some Gaussian noise.

Consider an $N\times D$ data matrix $X$ whose rows ${\mathbf{x}}_i\in {\mathbb{R}}^D$ are i.i.d. samples from an unknown distribution $p(\mathbf{x})$. PPCA assumes the samples are generated from a lower-dimensional latent features ${\mathbf{z}}_i\sim N(0,{\mathbf{I}}_K)\in {\mathbb{R}}^K$, where $K\ll D$.

The PPCA model is defined as ${\mathbf{x}}_i=\mathbf{W}{\mathbf{z}}_i+\mathbf{\mu }+{\mathbf{ϵ}}_i$ Where:

• $\mathbf{W}$ is a $D\times K$ loading matrix that maps the latent space to the observed space.
• $\mathbf{\mu }$ is the $D$-dimensional mean of the observed data.
• ${\mathbf{ϵ}}_i\sim N(0,{\sigma }^2{\mathbf{I}}_K)$ is the Gaussian noise term.

From the model, we can derive the properties of the observed data.

• $E[{\mathbf{x}}_i]=\mathbf{\mu }$
• $\mathrm{Cov}[{\mathbf{x}}_i]=\mathbf{W}{\mathbf{W}}^{⊺}+{\sigma }^2{\mathbf{I}}_K$

The goal of PPCA is to estimate the model parameters $\mathbf{W}$, $\mathbf{\mu }$, and ${\sigma }^2$ from the observed data. This is done using MLE, and the analytic form of MLE exists.

When the noise variance ${\sigma }^2$ is estimated using MLE, the principal subspace spanned by the columns of $\mathbf{W}$ is the same as the principal subspace of PCA.