Denoising Autoencoder

Definition

$D(x^{\prime })=g(f(x^{\prime }))=\approx x$ where $x^{\prime }=x+ϵ$, here $ϵ$ is a random noise, $f$ is an encoder, and $g$ is a decoder.

Denoising autoencoder is a type of Autoencoder designed to learn robust representations of data by reconstructing clean inputs from noisy versions.

Architecture

The model takes clean input $x$ and adds noise to create corrupted input $x^{\prime }$. The noisy input is fed into the model, and the model attempts to reconstruct the original clean input by minimizing MSE. $L(x,g(f(x^{\prime })))=||x-g(f(x^{\prime }))|{|}^2$

In a Gaussian noise setting, the estimator minimizing MSE is the mean of posterior distribution $E[x|x^{\prime }]$, so DAE learns a function that approximates the posterior mean $D(x^{\prime })\to E[x|x^{\prime }]$.

Tweedie’s Formula $\mathbb{E}[x|y]=y+{\sigma }^2\nabla \ln p(y)$ shows us that the DAE approximating $E[x|x^{\prime }]$ is actually the same task as estimating the Score Function of the true data distribution. $D(x^{\prime })\approx \mathbb{E}[x|x^{\prime }]=x^{\prime }+{\sigma }^2\underset{\text{score}}{\underbrace{\nabla \ln p(x^{\prime })}}$