Reward

Definition

Reward $R_t$ is a scalar feedback indicating how well the agent is doing at step $t$.

There are three reward types:

• $R_s=r(s)$: the reward in the state $s$.
• $R_s^a=r(s,a)$: the reward by the action $a$ in the state $s$.
• $R_{s{s}^{\prime }}^a=r(s,a,s^{\prime })$: the reward received after transitioning from state $s$ to $s^{\prime }$ due to action $a$.

Expected Reward

Under known dynamics $p(s^{\prime },r|s,a)$ of all transitions $(s,a,s^{\prime },r)$, one can compute the followings:

• State transition probability $P_{s{s}^{\prime }}^a=p(s^{\prime }|s,a)=P(S_{t+1}=s^{\prime }|S_t=s,A_t=a)=\sum _{r\in \mathcal{R}}p(s^{\prime },r|s,a)$

• Expected reward for (state, action) pair $R_s^a=r(s,a)=E[R_{t+1}|S_t=s,A_t=a]=\sum _{r\in \mathcal{R}}r\sum _{s^{\prime }\in \mathcal{S}}p(s^{\prime },r|s,a)$

• Expected reward for (state, action, next state) triple $R_{s{s}^{\prime }}^a=r(s,a,s^{\prime })=E[R_{t+1}|S_t=s,A_t=a,S_{t+1}=s^{\prime }]=\frac{\sum _{r\in \mathcal{R}}rp(s^{\prime },r|s,a)}{p(s^{\prime }|s,a)}$

Facts

Reinforcement learning is based on the Reward Hypothesis.