Policy Gradient Theorem

Definition

${\nabla }_{\theta }J(\theta )={\nabla }_{\theta }{E}_{{\pi }_{\theta }}[r(\tau )]=E_{{\pi }_{\theta }}\left[r(\tau ){\sum }_{t=0}^{T-1}{\nabla }_{\theta }\ln {\pi }_{\theta }(a_t|s_t)\right]$ The derivative of the expected total reward is the expectation of the product of total rewards and summed gradients of log of the policy ${\pi }_{\theta }$.

Proof

$\nabla E_{\pi }[r(\tau )]=\nabla \int r(\tau )\pi (\tau )d\tau =\int \frac{\nabla \pi (\tau )}{\pi (\tau )}r(\tau )\pi (\tau )d\tau =\int \nabla \ln (\pi (\tau ))r(\tau )\pi (\tau )d\tau =E[\nabla \ln (\pi (\tau ))r(\tau )]$ $\nabla \ln \pi (\tau )=\nabla \ln (p(s_0){\prod }_{t=0}^{T-1}\pi (a_t|s_t)p(s_{t+1}|s_t,a_t))=\sum _{t=0}^{T-1}\nabla \ln \pi (a_t|s_t)$