Momentum Optimizer

Definition

Momentum optimizer remembers the update at each iteration, and determines the next update as a linear combination of the gradient and the previous update

$\begin{array}{rl}{v}_{t+1}& =\rho v_t-\alpha \nabla f(x_t)\\ x_{t+1}& =x_t+v_{t+1}\end{array}$ where $0\le \rho \le 1$ is the momentum coefficient.