TD Error

Definition

The TD error $\delta$ is the difference between the target and the current prediction.

TD error for State-Value Function

${\delta }_t=r_{t+1}+\gamma V(s_{t+1})-V(s_t)$

TD error for SARSA

${\delta }_t=r_{t+1}+\gamma Q(s_{t+1},a_{t+1})-Q(s_t,a_t)$

TD error for Q-Learning

${\delta }_t=r_{t+1}+\gamma {\max }_{a}Q(s_{t+1},a)-Q(s_t,a_t)$