AdaGrad Optimizer

Definition

Adaptive gradient descent (AdaGrad) is a Gradient Descent with parameter-wise learning rate.

$g_t=\sum _{i=1}^t(\nabla f(x_i){)}^2$ $x_{t+1}=x_t-\alpha \frac{\nabla f(x_t)}{\sqrt{g_t+ϵ}}$ where $g_t$ is the sum of squares of past gradients, and $ϵ$ is a small constant to prevent division by zero.