Activation Function

Definition

The activation function of a node in an artificial neural network is a function that calculates the output of the node based on the linear combination of its inputs. It is used to add a non-linearity to the model.

Examples

Logistic Function

Definition

$\sigma (x)=\mathrm{logistic}(x)={\mathrm{logit}}^{-1}(x)=\frac{1}{1+\exp (-x)}$

The logistic function is inverse function of Logit.

Facts

Sigmoid activation function is vulnerable to vanishing gradient problem. $\frac{d}{dx}\sigma (x)=\sigma (x)(1-\sigma (x))$ The image of the derivative of the sigmoid function is $(0,0.25]$. For this reason, after passing node with sigmoid Activation Function, the gradient is decreased

Also, with the sigmoid Activation Function, if all the inputs are positive, then all the gradients also positive.

Link to original

Hyperbolic Tangent Function

Definition

$\tanh (x)=\frac{\sinh x}{\cosh x}=\frac{e^x-e^{-x}}{e^x+e^{-x}}=\frac{e^{2x}-1}{e^{2x}+1}$

Link to original

Rectified Linear Unit Function

Definition

$f(x)=\max (0,x)$

Facts

If an initial value is negative, it is never updated.

Link to original

ReLU6

Definition

$f(x)=\min (\max (0,x),6)$

Link to original

Gaussian-Error Linear Unit

Definition

GELU is a smooth approximation of ReLU.

$f(x)=x\Phi (x)$ where $\Phi$ is the CDF of the standard normal distribution.

Link to original

Parametric ReLU

Definition

$f(x)=\max (\alpha x,x)$ where $\alpha \le 1$ is a hyperparameter

Facts

If $\alpha =0.01$, it is called a Leaky ReLU

Link to original

Exponential Linear Unit

Definition

$f(x)=\{\begin{array}{ll}x& x\ge 0\\ \alpha (e^x-1)& x<0\end{array}$ where $\alpha \ge 0$ is a hyperparameter

Link to original

Swish Function

Definition

$f(x)=x\sigma (\beta x)=\frac{x}{1+e^{-\beta x}}$ where $\sigma$ is Sigmoid Function, and $\beta$ is a hyperparameter

When $\beta =1$, the function is called a sigmoid liniear unit (SiLU).

Link to original