Asynchronous Advantage Actor-Critic Method

Definition

A3C (Asynchronous Advantage Actor-Critic Method) is an Actor-Critic Method that utilizes multiple networks which are a global network and multiple worker agents working in parallel across multiple instances of the environment, and agents update asynchronously the global network parameter. The parallelism reduce each agent’s temporal correlation. The return $G_{t:t+n}-V(s_t)$ estimates the Advantage Function $Q(s_t,a_t)-V(s_t)$ using the n-Step Return $G_{t:t+n}$ instead of Action-Value Function $Q(s_t,a_t)$.

Actor (Policy Gradient Update)

$\Delta \theta =\alpha [(G_{t:t+n}-V(s_t;\varphi )){\nabla }_{\theta }\ln {\pi }_{\theta }(a_t|s_t)+\lambda H({\pi }_{\theta }(a_t|s_t))]$ where $H(p(x))=-\int p(x)\ln p(x)dx$ is an Entropy that encourages exploration

Critic (Value Network Update)

$\Delta \varphi =\beta (G_{t:t+n}-V(s_t;\varphi )){\nabla }_{\varphi }V(s_t;\varphi )$

Algorithm

For each worker agent

1. Let $\theta ,\varphi$ be global network’s shared parameters, and let ${\theta }^{\prime },{\varphi }^{\prime }$ be a worker agent’s parameters.
2. Set the hyperparameters: step-sizes $\alpha ,\beta >0$, discount factor $0<\gamma \le 1$, regularization factor $\lambda$, maximum steps per update $t_{\text{max}}$.
3. Repeat:
   1. Reset gradients $d\theta \leftarrow 0$ and $d\varphi \leftarrow 0$.
   2. Synchronize parameters ${\theta }^{\prime }\leftarrow \theta$ and $\varphi \leftarrow {\varphi }^{\prime }$.
   3. Set $t_{\text{start}}\leftarrow t$ and get state $s_t$.
   4. For $t\le t_{\text{start}}+t_{\text{max}}$:
      1. Select action $a_t$ according to policy $\pi (\cdot |s_t;{\theta }^{\prime })$.
      2. Take the action $a_t$ and observe a reward $r_{t+1}$ and a next state $s_{t+1}$.
      3. $t\leftarrow t+1$
   5. $R=V(s_t;{\varphi }^{\prime })$ (or $0$ for terminal $s_t$)
   6. For $i=t-1,t-2,\dots ,t_{\text{start}}$:
      1. $R\leftarrow r_{i+1}+\gamma R$.
      2. Accumulate gradients with respect to ${\theta }^{\prime }$ $d\theta \leftarrow d\theta +(R-V(s_i;{\varphi }^{\prime })){\nabla }_{{\theta }^{\prime }}\ln {\pi }_{{\theta }^{\prime }}(a_i|s_i)+\lambda H({\pi }_{\theta }(a_i|s_i))]$
      3. Accumulate gradients with respect to ${\varphi }^{\prime }$ $d\varphi \leftarrow d\varphi -(R-V(s_i;{\varphi }^{\prime })){\nabla }_{{\varphi }^{\prime }}V(s_t;{\varphi }^{\prime })$
   7. Update asynchronously $\theta \leftarrow \theta +\alpha d\theta$ and $\varphi \leftarrow \varphi -\beta d\varphi$.