Factor Analysis

Definition

Factor Analysis (FA) is a dimensionality reduction techniques that that aims to explain the covariance structure of given data using a fewer number of latent variables (features). FA models the observed data as linear combinations of latent features plus some Gaussian noise.

Consider an $N\times D$ data matrix $X$ whose rows ${\mathbf{x}}_i\in {\mathbb{R}}^D$ are i.i.d. samples from an unknown distribution $p(\mathbf{x})$. FA assumes the samples are generated from a lower-dimensional latent features ${\mathbf{f}}_i\sim N(0,{\mathbf{I}}_K)\in {\mathbb{R}}^K$, where $K\ll D$.

The FA model is defined as ${\mathbf{x}}_i=\mathbf{W}{\mathbf{f}}_i+\mathbf{\mu }+{\mathbf{\psi }}_i$ Where:

• $\mathbf{W}$ is a $D\times K$ factor loading matrix that maps the latent space to the observed space.
• $\mathbf{\mu }$ is the $D$-dimensional mean of the observed data.
• ${\mathbf{\psi }}_i\sim N(0,\mathbf{\Psi })$ is the Gaussian noise term, where $\mathbf{\Psi }$ is a $D\times D$ diagonal matrix.

From the model, we can derive the properties of the observed data.

• $E[{\mathbf{x}}_i]=\mu$
• $\mathrm{Cov}[{\mathbf{x}}_i]=\mathbf{W}{\mathbf{W}}^{⊺}+\mathbf{\Psi }$

The goal of FA is to estimate the factor loading matrix $\mathbf{W}$, $\mathbf{\mu }$, and the variance matrix $\mathbf{\Psi }$ from the observed data. This is done using either SVD-based approach or EM Algorithm.

Each observed variable has its own variance (Since $\mathbf{\Psi }$ is diagonal), representing the variability not explained by the common factors. This is a crucial difference from PPCA.

In FA, the log-likelihood function with respect to the parameters is invariant to orthogonal transformations of $\mathbf{W}$. That is, replacing $\mathbf{W}$ with $\mathbf{WR}$ where $\mathbf{R}$ is any Orthonormal Matrix does not change the value. The rotational freedom can be used to achieve a more interpretable factor structure.

PPCA is a special case of Factor Analysis which restricts $\mathbf{\Psi }$ to have the same value along the diagonal $\mathbf{\Psi }={\sigma }^2{\mathbf{I}}_D$.