Double DQN

Definition

Double DQN is a variation of DQN that uses the idea of Double Q-Learning to reduce overestimation. In this case, the target network in DQN works as the second network for double Q-learning without introducing an additional network.

$L(\theta )=[r_{t+1}+\gamma \hat{Q}(s_{t+1},\underset{a}{\mathrm{argmax}}Q(s_{t+1},a;\theta );\hat{\theta })-Q(s_t,a_t;\theta ){]}^2$

Algorithm

1. Initialize behavior network $Q$ and target network $\hat{Q}$ with random weights $\theta$, and the replay buffer $\mathcal{R}$ to max size $N$.
2. Repeat for each episode:
   1. Initialize sequence $s_0$.
   2. Repeat for each step of an episode until terminal, $t=0,1,\dots ,T-1$ :
      1. With probability $ϵ$, select a random action $a_t$ otherwise select $a_t=\underset{a}{\mathrm{argmax}}Q(s_t,a;\theta )$.
      2. Take the action $a_t$ and observe a reward $r_{t+1}$ and a next state $s_{t+1}$.
      3. Store transition $(s_t,a_t,r_{t+1},s_{t+1})$ in the replay buffer $\mathcal{R}$.
      4. Sample random minibatch of $B$ transitions $(s_i,a_i,r_{i+1},s_{i+1})$ from $\mathcal{R}$.
      5. $y_i\leftarrow r_{i+1}+\gamma \hat{Q}(s_{i+1},\underset{a}{\mathrm{argmax}}Q(s_{i+1},a;\theta );\hat{\theta })$.
      6. Perform Gradient Descent on loss $L(\theta )=(y_i-Q(s_i,a_i;\theta ){)}^2$.
      7. Update the target network parameter $\hat{\theta }\leftarrow \theta$ every $C$ steps.
      8. Update $s_t\leftarrow s_{t+1}$.