Overdispersion

Definition

Overdispersion is the presence of greater observed variance than what would be expected under a given statistical model. It can happen due to heterogeneity or lack of independence between trials, and clustering or grouping in the data.

Overdispersion on Binomial Distribution

Assume that there are $k$ clusters, the number of observations of each cluster is $m$. Let the Random Variable following Binomial Distribution $X_i\sim B(m,{\pi }_i)$ be the number of successes out of $m$ observation, where ${\pi }_i$ is also Random Variable with mean $E({\pi }_i)=\pi$ and variance $\mathrm{Var}({\pi }_i)={\sigma }^2\pi (1-\pi )$. Also, let $Y=\sum _{i=1}^k{X}_i\sim B(n,\pi )$ be the total number of successes in the all clusters, where $m:=mk$. Then, the mean and variance of $Y$ is defined as $E(Y)=n\pi$ $Var(Y)=n\pi (1-\pi )[1+(m-1){\sigma }^2]$ Here, $\varphi =1+(m-1){\sigma }^2$ is called a dispersion parameter. If $\varphi >1$, then overdispersion occurred.

The Lexis Ratio is also used for detecting overdispersion $Q=\frac{{\sum }_{i=1}^k({\hat{\pi }}_i-\hat{\pi }{)}^2/(k-1)}{\hat{\pi }(1-\hat{\pi })/n}$ where ${\hat{\pi }}_i=\frac{X_i}{m}$ and $\hat{\pi }=\frac{Y}{n}$

Overdispersion on Poisson Distribution

If $\mathrm{Var}(Y)>E(Y)$ in Poission Distribution setting, then the overdispersion might occur. Let i.i.d. $Z_i$ be the number of occurrences at the $i$-th cluster, and the number of clusters $N$ is also a Random Variable. Also, let $Y=\sum _{i=1}^N{Z}_i$ be the total number of occurrences in $N$ clusters, where $N$ follows a Poission Distribution and independent to $Z$. Then, the mean and variance of $Y$ is defined as $E(Y)=E(N)E(Z)$ $\mathrm{Var}(Y)=E(N)E(Z^2)$ If $E(Z^2)\gg E(Z)$, then the overdispersion occurs.