Swish Function

Definition

$f(x)=x\sigma (\beta x)=\frac{x}{1+e^{-\beta x}}$ where $\sigma$ is Sigmoid Function, and $\beta$ is a hyperparameter

When $\beta =1$, the function is called a sigmoid liniear unit (SiLU).