Mutual Information

Definition

$I(X;Y)=D_{KL}(P(X,Y)||P(X)P(Y))=H(X)-H(X|Y)$

Mutual information of two random variables quantifies the amount of information obtained about one Random Variable by observing the other Random Variable.