Kullback-Leibler Divergence

Definition

Assume that two Probability Distributions $P$ and $Q$ are given. Then the Kullback-Leibler divergence between $P$ and $Q$ is defined as $D_{KL}(P||Q)=E_{x\sim P}\left[\ln \left(\frac{p(x)}{q(x)}\right)\right]={\int }_{x}p(x)\ln \frac{p(x)}{q(x)}dx$

Kullback-Leibler divergence measures how different two distributions are.

It also can be expressed as a difference between the cross entropy $H(p,q)$ (difference between distributions $p$ and $q$)and entropy $H(p)$ (inherent uncertainty of $p$). $H(P,Q)-H(P)={\int }_{x}p(x)\ln \frac{1}{q(x)}dx+{\int }_{x}p(x)\ln p(x)dx={\int }_{x}p(x)\ln \frac{p(x)}{q(x)}dx=D_{KL}(P\Vert Q)$

Facts

Let $\{P_n{\}}_{n\in \mathbb{N}}$ be a sequence of distributions. Then, $D_{KL}(P_n||P)\to 0⟹D_{JS}(P_n,P)\to 0⟺\delta (P_n,P)\to 0⟹W(P_n,P)\to 0⟺P_n\stackrel{D}{\to }P$ The convergence of the KL-Divergence to zero implies that the JS-Divergence also converges to zero. The convergence of the JS-Divergence to zero is equivalent to the convergence of the Total Variation Distance to zero. The convergence of the Total Variation Distance to zero implies that the Wasserstein Distance also converges to zero. The convergence of the Wasserstein Distance to zero is equivalent to the Convergence in Distribution of the sequence.