Delta Method

Definition

Univariate Delta Method

Let $\{X_n\}$ be a sequence of random variables satisfying $\sqrt{n}(X_n-\theta )\stackrel{D}{\to }N(0,{\sigma }^2)$, $g$ be a differentiable function at $x=\theta$, and $g^{\prime }(\theta )\ne 0$, then $\sqrt{n}(g(X_n)-g(\theta ))\stackrel{D}{\to }{g}^{\prime }(\theta )N(0,{\sigma }^2)=N(0,{\sigma }^2{g}^{\prime }(\theta {)}^2)$

Proof

By Taylor Series approximation $g(X_n)=g(\theta )+g^{\prime }(\theta )(X_n-\theta )+o_P(|X_n-\theta |)$ $\Rightarrow \sqrt{n}(g(X_n)-g(\theta ))=g^{\prime }(\theta )\sqrt{n}(X_n-\theta )+o_P(\sqrt{n}|X_n-\theta |)$

Where $\sqrt{n}(X_n-\theta )\stackrel{D}{\to }N(0,{\sigma }^2)$ by the assumption. By the continuous mapping theorem, $\sqrt{n}|X_n-\theta |$, which is a function of $X_n$, also converges to random variable, so it is Boundedness in Probability $\sqrt{n}|X_n-\theta |=O_P(1)$ by the property of converging random vector

Therefore, $\sqrt{n}(g(X_n)-g(\theta ))=g^{\prime }(\theta )N(0,{\sigma }^2)+o_P(O_P(1))=g^{\prime }(\theta )N(0,{\sigma }^2)=N(0,{\sigma }^2{g}^{\prime }(\theta {)}^2)$ by the property of sequence of random variables bounded in probability

Examples

Estimation of the sample variance of Bernoulli Distribution

Let $X_1,X_2,\dots ,X_n\sim Ber(p)$ $\sqrt{n}({\bar{X}}_n-p)\to N(0,p(1-p))$ by CLT

$\sqrt{n}(g({\bar{X}}_n)-g(p))\to g^{\prime }(p)N(0,p(1-p))$ by Delta method

Let $g(x)=x(1-x)$, then $\sqrt{n}({\bar{X}}_n(1-{\bar{X}}_n)-p(1-p))\to N(0,p(1-p)(1-2p{)}^2)$

Therefore, the sample mean and variance follow such distributions ${\bar{X}}_n\to N\left(p,\frac{p(1-p)}{n}\right)=N\left(\mu ,\frac{\sigma }{n}\right)$ ${\bar{X}}_n(1-{\bar{X}}_n)=\hat{\sigma }\to N\left(p(1-p),\frac{p(1-p)(1-2p{)}^2}{n}\right)=N\left(\sigma ,\frac{\sigma (1-2p{)}^2}{n}\right)$

Visualization

• x-axis: $\mu =p$
• y_axis: ${\sigma }^2=p(1-p)$
• y1: $y=p(1-p)$^[variance by mean]
• y2: ${\bar{X}}_n$^[sample mean]
• y3, y4: ${\bar{X}}_n(1-{\bar{X}}_n)$^[sample variance that calculated by the sample mean]
• y5: first order approximated line at $(p,p(1-p))$

If sample size $n\to \infty$, variance of sample mean $V({\bar{X}}_n)\to 0$. So, sample variance $\hat{\sigma }$ can be well approximated by first-order approximation.