Exponential Family

Definition

General Representation

$p(x|\mathbf{\theta })-\frac{1}{Z(\mathbf{\theta })}h(\mathbf{x})\exp (\mathbf{t}(\mathbf{x})\cdot \mathbf{\theta })$ Where:

• $\mathbf{\theta }$ is the vector of natural parameters.
• $\mathbf{t}(\mathbf{x})$ is the vector of sufficient statistics.
• $h(\mathbf{x})$ is the non-negative volume of $x$
• $Z(\mathbf{\theta })=\int h(\mathbf{x})\exp (\mathbf{t}(\mathbf{x})\cdot \mathbf{\theta })\nu (d\mathbf{x})$ is the normalizer $\nu (d\mathbf{x})$ refers to the measure of $\mathbf{x}$.

Canonical Form with Dispersion Parameter

$f(x|\theta ,\varphi )=\exp \left[\frac{x\theta -b(\theta )}{a(\varphi )}+c(x,\varphi )\right]$ where $a(\cdot )$, $b(\cdot )$, and $c(\cdot )$ are known functions, $V(\mu ):=b^{\prime \prime }(\theta )$ is called a variance function, and $\varphi$ is called a dispersion parameter. This form is mainly used for GLMs.

Examples

Distributions	Normal	Poisson	Binomial	Gamma	Inverse Gaussian
Notation	$N(\mu ,{\sigma }^2)$	$\mathrm{Pois}(\lambda )$	$B(n,\pi )$	$\Gamma (\alpha ,\beta )$	$IG(\mu ,\lambda )$
$a(\varphi )$	$\varphi$	$1$	$1/n$	$\varphi$	$\varphi$
$b(\theta )$	${\theta }^2/2$	$e^{\theta }$	$\ln (1+e^{\theta })$	$-\ln (-\theta )$	$-(-2\theta {)}^{1/2}$
$c(x,\varphi )$	$-\frac{1}{2}[\ln (2\pi {\sigma }^2)+\frac{x^2}{{\sigma }^2}]$	$-\ln (x!)$	$\ln \left(\genfrac{}{}{0ex}{}{n}{x}\right)$	$\alpha \ln (\alpha x)-\ln \Gamma (\alpha )-\alpha \ln (x)$	$-\frac{1}{2}[\ln (2\pi \lambda x^3)+\frac{1}{\lambda x}]$
$\mu (\theta )=b^{\prime }(\theta )$	$\theta$	$e^{\theta }$	$e^{\theta }/(1+e^{\theta })$	$-1/\theta$	$(-2\theta {)}^{-1/2}$
$V(\mu )$	$1$	$\mu$	$\mu (1-\mu )$	${\mu }^2$	${\mu }^3$
$\varphi$	${\sigma }^2$	$1$	$1/n$	$1/\alpha$	$\lambda$
Natural link	Identity	log	logit	inverse	${\mu }^{-2}$

Facts

For an exponential family, $E(X)=\mu =b^{\prime }(\theta )$ and $\mathrm{Var}(X)=b^{\prime \prime }(\theta )a(\varphi )$ holds by Bartlett Identities