Information Theory Note

Entropy

Definition

$H(X)=-E_{x\sim p(x)}[\log p(x)]=-{\int }_{x}p(x)\log p(x)dx$

The entropy of a Random Variable quantifies the average level of uncertainty or information associated with the variable’s potential states or possible outcomes.

Link to original

Joint Entropy

Definition

$H(X,Y)=-E_{(X,Y)\sim p(x,y)}[\log (p(x,y))]=-{\int }_{X,Y}p(x,y)\log p(x,y)dxdy$

Joint entropy is a measure of the uncertainty associated with a set of random variables.

Link to original

Conditional Entropy

Definition

$H(Y|X)=H(X,Y)-H(X)=-E_{(X,Y)\sim p(x,y)}\left[\log \left(\frac{p(x,y)}{p(x)}\right)\right]=-{\int }_{X,Y}p(x,y)\log p(y|x)dxdy$

Conditional entropy quantifies the amount of information needed to describe the outcome of a Random Variable $Y$ given that the value of another Random Variable $X$ is known.

Link to original

Mutual Information

Definition

$I(X;Y)=D_{KL}(P(X,Y)||P(X)P(Y))=H(X)-H(X|Y)$

Mutual information of two random variables quantifies the amount of information obtained about one Random Variable by observing the other Random Variable.

Link to original