Action-Value Function

Definition

$q_{\pi }(s,a)=E_{\pi }[G_t|S_t=s,A_t=a]$ An action-value function $q_{\pi }(s,a)$ is the expected discounted Return of taking action $a$ starting in state $s$ under policy $\pi$.

Optimal Action-Value Function

$q_{\ast }(s,a)=q_{{\pi }_{\ast }}(s,a)={\max }_{\pi }{q}_{\pi }(s,a)=r(s,a)+\gamma \sum _{s^{\prime }}p(s^{\prime }|s,a)v_{\ast }(s^{\prime })$ An optimal action-value function is the maximum possible action-value function (an action-value function under optimal policy).