State Visitation Frequency

Definition

State visitation frequency is the discounted sum of probabilities of visiting a given state $s$ under policy $\pi$. ${\rho }_{\pi }(s)=\sum _{t=0}^{\infty }{\gamma }^{t}P(s_t=s|\pi )={\int }_S{\sum }_{t=0}^{\infty }{\gamma }^t{p}_0(s^{\prime })p_t(s^{\prime }\to s|\pi )d{s}^{\prime }$ where $p_0(s^{\prime })$ is the initial and $p_t(s^{\prime }\to s|\pi )$ is the visitation probability from $s^{\prime }$ to $s$ in $t$ step following $\pi$.

$(1-\gamma ){\rho }_{\pi }$ can be considered as a Distribution Function over the state space.