Jensen-Shannon Divergence

Definition

Jensen–Shannon divergence is a method of measuring the similarity between two Probability Distributions defined by KL-Divergence

Assume that two Probability Distributions $P$ and $Q$ are given. Then the Jensen–Shannon divergence between $P$ and $Q$ is defined as $D_{JS}(P,Q)=\frac{1}{2}{D}_{KL}(P||\frac{P+Q}{2})+\frac{1}{2}{D}_{KL}(Q||\frac{P+Q}{2})$ where $D_{KL}$ is the KL-Divergence

Facts

Let $\{P_n{\}}_{n\in \mathbb{N}}$ be a sequence of distributions. Then, $D_{KL}(P_n||P)\to 0⟹D_{JS}(P_n,P)\to 0⟺\delta (P_n,P)\to 0⟹W(P_n,P)\to 0⟺P_n\stackrel{D}{\to }P$ The convergence of the KL-Divergence to zero implies that the JS-Divergence also converges to zero. The convergence of the JS-Divergence to zero is equivalent to the convergence of the Total Variation Distance to zero. The convergence of the Total Variation Distance to zero implies that the Wasserstein Distance also converges to zero. The convergence of the Wasserstein Distance to zero is equivalent to the Convergence in Distribution of the sequence.

Link to original