Adam Optimizer

Definition

Adaptive momentum estimation (Adam) combines the ideas of momentum and RMSProp optimizers.

$\begin{array}{rl}{m}_t& ={\beta }_1{m}_{t-1}+(1-{\beta }_1)\nabla f(x_t)\\ v_t& ={\beta }_2{v}_{t-1}+(1-{\beta }_2)(\nabla f(x_t){)}^2\\ {\hat{m}}_t& =\frac{m_t}{1-{\beta }_1^t}\\ {\hat{v}}_t& =\frac{v_t}{1-{\beta }_2^t}\\ x_{t+1}& =x_t-\alpha \frac{{\hat{m}}_t}{\sqrt{{\hat{v}}_t}+ϵ}\end{array}$ Where:

• $m_t$ is the estimate of the first moment (mean) of the gradients
• $v_t$ is the estimate of the second moment (un-centered variance) of the gradients
• ${\beta }_1$ and ${\beta }_2$ are decay rates for the moment estimates
• ${\hat{m}}_t$ and ${\hat{v}}_t$ are bias-corrected estimates