Actor-Critic Method with TD(0) Return

Definition

Actor (Policy Gradient Update)

$\Delta \theta =\alpha (r_{t+1}+\gamma V(s_{t+1};\varphi )-V(s_t;\varphi )){\nabla }_{\theta }\ln {\pi }_{\theta }(a_t|s_t)$

Critic (Value Network Update)

$\Delta \varphi =\beta (r+\gamma V(s_{t+1};\varphi )-V(s_t;\varphi )){\nabla }_{\varphi }V(s_t;\varphi )$

Algorithm

1. Initialize critic $V(s;\varphi )$ and actor $\pi (a|s;\theta )$ networks randomly.
2. Set the hyperparameters: step-sizes $\alpha ,\beta >0$, and discount factor $0<\gamma \le 1$
3. Repeat for each episode (Each starts from a state $s$ under the policy ${\pi }_{\theta }$):
   1. $I=1$
   2. Repeat for each step of an episode until terminal, $t=0,1,\dots ,T-1$ :
      1. Select action $a$ according to policy $\pi (\cdot |s;\theta )$.
      2. Take the action $a$ and observe a reward $r$ and a next state $s^{\prime }$.
      3. $\delta \leftarrow r+\gamma V(s^{\prime };\varphi )-V(s;\varphi )$
      4. $\varphi \leftarrow \varphi +\beta \delta {\nabla }_{\varphi }V(s;\varphi )$
      5. $\theta \leftarrow \theta +\alpha I\delta {\nabla }_{\theta }\ln \pi (a|s;\theta )$
      6. $I\leftarrow \gamma I$
      7. $s\leftarrow s^{\prime }$