Gaussian-Error Linear Unit

Definition

GELU is a smooth approximation of ReLU.

$f(x)=x\Phi (x)$ where $\Phi$ is the CDF of the standard normal distribution.