State-Value Function

Definition

$v_{\pi }(s)=E_{\pi }[G_t|S_t=s]=\sum _a\pi (a|s)q_{\pi }(s,a)$ where $q_{\pi }(s,a)$ is a Action-Value Function.

A state-value function $v_{\pi }(s)$ is the expected discounted Return starting in state $s$ under policy $\pi$.

Optimal State-Value Function

$v_{\ast }(s)=v_{{\pi }_{\ast }}(s)={\max }_{\pi }{v}_{\pi }(s)={\max }_a{q}_{\ast }(s,a)$ An optimal state-value function is the maximum possible state-value function (a state-value function under optimal policy).