Off-Policy Control

Definition

Off-policy control uses two different policies for target policy $\pi$ and behavior policy $\mu$. The target policy $\pi$ may be deterministic while the behavior policy $\mu$ is stochastic. It enables the model to learn optimal policy while maintaining exploration.

Almost all off-policy method utilize Importance Sampling. The Return obtained under the target policy is weighted according to the relative probability of the trajectories under the target and behavior policy, called the importance-sampling ratio (importance weight).

Given a starting state $S_t$, the probability of the state-action trajectory $\{A_t,S_{t+1},A_{t+1},\dots ,S_T\}$ occurring under a policy $\pi$ is written as $P(\{A_t,S_{t+1},A_{t+1},\dots ,S_T\}|S_t,A_{t:T-1}\sim \pi )={\prod }_{k=t}^{T-1}\pi (A_k|S_k)p(S_{k+1}|S_k,A_k)$ where $p$ is the state-transition probability.

Thus, the relative probability of the trajectory under the target and behavior policies is ${\rho }_{t:T-1}={\prod }_{k=t}^{T-1}\frac{\pi (A_k|S_k)}{\mu (A_k|S_k)}$

Algorithms

Off-policy MC

$Q(S_t,A_t)\leftarrow Q(S_t,A_t)+\alpha \left[{\rho }_{t:T-1}{G}_t-Q(S_t,A_t)\right]$ where ${\rho }_{t:T-1}$ is the importance weight.

The variance of return in Monte Carlo method can dramatically higher.

Off-policy Sarsa

$Q(S_t,A_t)\leftarrow Q(S_t,A_t)+\alpha \left[\frac{\pi (A_{t+1}|S_{t+1})}{\mu (A_{t+1}|S_{t+1})}(R_{t+1}+\gamma Q(S_{t+1},A_{t+1}))-Q(S_t,A_t)\right]$

The variance of return in off-policy TD is lower than MC’s one.