Optimizers

Definition

Optimizers are algorithms used to adjust the parameters of a model to minimize the loss function. The optimizers aim to improve the convergence speed and stability of the training process compared to standard Stochastic Gradient Descent.

Examples

Momentum Optimizer

Definition

Momentum optimizer remembers the update at each iteration, and determines the next update as a linear combination of the gradient and the previous update

$\begin{array}{rl}{v}_{t+1}& =\rho v_t-\alpha \nabla f(x_t)\\ x_{t+1}& =x_t+v_{t+1}\end{array}$ where $0\le \rho \le 1$ is the momentum coefficient.

Link to original

AdaGrad Optimizer

Definition

Adaptive gradient descent (AdaGrad) is a Gradient Descent with parameter-wise learning rate.

$g_t=\sum _{i=1}^t(\nabla f(x_i){)}^2$ $x_{t+1}=x_t-\alpha \frac{\nabla f(x_t)}{\sqrt{g_t+ϵ}}$ where $g_t$ is the sum of squares of past gradients, and $ϵ$ is a small constant to prevent division by zero.

Link to original

RMSProp Optimizer

Definition

RMSProp optimizer resolves AdaGrad Optimizer’s rapidly diminishing learning rates and relative magnitude difference by taking the exponential moving average on history.

$g_t=\gamma g_{t-1}+(1-\gamma )(\nabla f(x_t){)}^2$ $x_{t+1}=x_t-\alpha \frac{\nabla f(x_t)}{\sqrt{g_t+ϵ}}$ where $\gamma$ is the decay rate.

Link to original

Adam Optimizer

Definition

Adaptive momentum estimation (Adam) combines the ideas of momentum and RMSProp optimizers.

$\begin{array}{rl}{m}_t& ={\beta }_1{m}_{t-1}+(1-{\beta }_1)\nabla f(x_t)\\ v_t& ={\beta }_2{v}_{t-1}+(1-{\beta }_2)(\nabla f(x_t){)}^2\\ {\hat{m}}_t& =\frac{m_t}{1-{\beta }_1^t}\\ {\hat{v}}_t& =\frac{v_t}{1-{\beta }_2^t}\\ x_{t+1}& =x_t-\alpha \frac{{\hat{m}}_t}{\sqrt{{\hat{v}}_t}+ϵ}\end{array}$ Where:

• $m_t$ is the estimate of the first moment (mean) of the gradients
• $v_t$ is the estimate of the second moment (un-centered variance) of the gradients
• ${\beta }_1$ and ${\beta }_2$ are decay rates for the moment estimates
• ${\hat{m}}_t$ and ${\hat{v}}_t$ are bias-corrected estimates

Link to original