REINFORCE

Definition

${\nabla }_{\theta }J(\theta )\approx \frac{1}{M}\sum _{i=1}^M\left[{\sum }_{t=0}^{T-1}{G}_t^{(i)}{\nabla }_{\theta }\ln {\pi }_{\theta }(a_t^{(i)}|s_t^{(i)})\right]$ REINFORCE algorithm is a policy gradient algorithm that maximizes the expected return. The objective function of REINFORCE algorithm based on the Policy Gradient Theorem. It substitutes the expectation and the total reward of Policy Gradient with averaging and returns $G_t$.

Algorithm

1. Execute $M$ trajectories (Each starts from a state $s$ under the policy ${\pi }_{\theta }$).
2. Approximate the gradient of the objective function $J(\theta )$ $g_{\theta }\leftarrow \frac{1}{M}\sum _{i=1}^M\left[{\sum }_{t=0}^{T-1}{G}_t^{(i)}{\nabla }_{\theta }\ln {\pi }_{\theta }(a_t^{(i)}|s_t^{(i)})\right]$
3. Update policy to maximize $J(\theta )$ $\theta \leftarrow \theta +\alpha g_{\theta }$ where $\alpha$ is a learning rate.