Actor-Critic Method

Definition

Actor-Critic method consists of two networks: actor and critic networks.

Actor network updates parameter $\theta$ for policy ${\pi }_{\theta }(a|s)$ by maximizing $J(\theta )$ using Policy Gradient ${\nabla }_{\theta }J(\theta )=E_{{\pi }_{\theta }}\left[{\sum }_{t=0}^{T-1}\text{target}{\nabla }_{\theta }\ln {\pi }_{\theta }(a_t|s_t)\right]$

Critic network updates parameter $\varphi$ for value function $V(s;\varphi )$ by minimizing $L(\varphi )=E_{\pi }[{\sum }_{t=0}^{T-1}(\text{target}-V(s_t;\varphi ){)}^2]$

Examples

REINFORCE with Baseline

Actor-Critic Method with TD(0) Return

Asynchronous Advantage Actor-Critic Method

Deterministic Policy Gradient

Deep Deterministic Policy Gradient