RMSProp Optimizer

Definition

RMSProp optimizer resolves AdaGrad Optimizer’s rapidly diminishing learning rates and relative magnitude difference by taking the exponential moving average on history.

$g_t=\gamma g_{t-1}+(1-\gamma )(\nabla f(x_t){)}^2$ $x_{t+1}=x_t-\alpha \frac{\nabla f(x_t)}{\sqrt{g_t+ϵ}}$ where $\gamma$ is the decay rate.