Policy Gradient

Definition

Policy gradient algorithms directly learn the optimal policy by a parametric probability distribution ${\pi }_{\theta }(a|s)$. The policy stochastically selects an action $a$ in a state $s$ according to a parameter $\theta$. It typically proceeds by sampling a stochastic policy and adjusting the parameter $\theta$ in the direction of maximizing the total reward.

The objective function of policy gradient algorithm is defined as: $J(\theta )=E_{{\pi }_{\theta }}[r(\tau )]=\int p(\tau ;\theta )r(\tau )d\tau$ where $\tau$ is a trajectory, $r(\tau )$ is the total reward of the trajectory $\tau$, and $p(\tau ;\theta )={\pi }_{\theta }(\tau )$ is the PDF of $\tau$.