Batch Normalization

Definition

Batch normalization (batch norm) make training of neural network faster through normalization of each layers’ input by re-centering and re-scaling $BN(x)=\gamma \frac{x-{\mu }_{\text{batch}}}{\sqrt{{\sigma }_{\text{batch}}^2+ϵ}}+\beta$ where:

• ${\mu }_{\text{batch}}=\frac{1}{B}\sum _{i=1}^B{x}_i$ is the mean vector of the mini-batch
• ${\sigma }_{\text{batch}}^2=\frac{1}{B}\sum _{i=1}^B(x_i-{\mu }_{\text{batch}}{)}^2$ is the variance vector of the mini-batch
• $B$ is the batch size.
• $\gamma$, $\beta$ are the learnable scaling parameters.
• $ϵ$ is a small constant to prevent division by zero.

In test stage, the training stage’s moving average of the ${\mu }_{\text{batch}}$ and ${\sigma }_{\text{batch}}$ are used.