n-step Off-Policy Learning

Definition

The relative probability of the n-step trajectory under the target and behavior policies is ${\rho }_{t:t+n}={\prod }_{k=t}^{t+n}\frac{\pi (A_k|S_k)}{\mu (A_k|S_k)}$

Algorithms

Off-policy n-Step TD

$V(S_t)\leftarrow V(S_t)+\alpha [{\rho }_{t:t+n}{G}_{t:t+n}-V(S_t)]$ where ${\rho }_{t:t+n}$ is the importance weight.

Off-policy n-Step Sarsa

$Q(S_t,A_t)\leftarrow Q(S_t,A_t)+\alpha [{\rho }_{t:t+n}{G}_{t:t+n}-Q(S_t,A_t)]$ where ${\rho }_{t:t+n}$ is the importance weight, and $G_{t:t+n}:=\sum _{k=0}^{n-1}{\gamma }^k{R}_{t+k+1}+{\gamma }^{n}Q(S_{t+n},A_{t+n})$.