Gibbs Sampling

Definition

A MCMC algorithm for sampling from a specified Multivariate Distribution when direct sampling from the joint distribution is difficult, but sampling from the Conditional Distribution is more practical.

Algorithm

1. Begin with some initial value $X^{(0)}$
2. For $i=1,\dots k$ generate $X^{(i+1)}$ from $p(X_j^{(i+1)}|X_1^{(i+1)},\dots ,X_{j-1}^{(i+1)},X_{j+1}^{(i+1)},\dots ,X_n^{(i+1)})$
3. Repeat the above step $k$ times.

Examples

Consider a bivariate random variable $(X,Y)$, and suppose we want to obtain samples from the marginals $f(x),f(y)$. Take samples from the conditional distributions $x_i\sim f(x|y=y_{i-1})$ and $y_i\sim f(y|x=x_{i-1})$. As $k\to \infty$, the samples $x_k,y_k$ are eventually samples from $f(x)$ and $f(y)$ respectively.

Facts

It is common to discard some number of samples at the beginning (burn-in period).