Maximum Likelihood Estimation

Definition

MLE is the method of estimating the parameters of an assumed Distribution

Let $X_1,X_2,\dots ,X_n$ be Random Sample with PDF $f(x|\theta )$, where $\theta \in \Omega$, then the MLE ${\hat{\theta }}_{\text{MLE}}$ of $\theta$ is estimated as ${\hat{\theta }}_{\text{MLE}}=\underset{\theta }{\mathrm{argmax}}L(\theta |\mathbf{x})$

Regularity Conditions

• R0: The pdfs are distinct, i.e. $\theta \ne {\theta }^{\prime }\Rightarrow f(x_i|\theta )\ne f(x_i|{\theta }^{\prime })$
• R1: The pdfs have same supports $\forall \theta$
• R2: The true value ${\theta }_0$ is an interior point in $\Omega$
• R3: The pdf $f(x|\theta )$ is twice differentiable with respect to $\theta \in \Omega$
• R4: $\frac{\partial }{\partial {\theta }^2}\int f(x|\theta )dx=\int \frac{\partial }{\partial {\theta }^2}f(x|\theta )dx$
• R5: The pdf $f(x|\theta )$ is three times differentiable with respect to $\theta \in \Omega$, $\forall \theta \in \Omega ,\left|\frac{{\partial }^3}{\partial {\theta }^3}\ln f(x|\theta )\right|\le M(x)$, and $\exists c\in \mathbb{R},\exists M(x),\forall |\theta -{\theta }_0|<c,\forall$ interior point $x,E_{{\theta }_0}[M(X)]<\infty$

Properties

Functional Invariance

If $\hat{\theta }$ is the MLE for $\theta$, then $g(\hat{\theta })$ is the MLE of $g(\theta )$

Consistency

Under R0 ~ R2 Regularity Conditions, let ${\theta }_0$ be a true parameter, $f(x|\theta )$ is differentiable with respect to $\theta \in \Omega$, then $\frac{\partial }{\partial \theta }L(\theta )=0$ has a solution ${\hat{\theta }}_n$ such that ${\hat{\theta }}_n\stackrel{P}{\to }{\theta }_0$

Asymptotic Normality

Under the R0 ~ R5 Regularity Conditions, let $X_1,X_2,\dots ,X_n$ be Random Sample with PDF $f(x|\theta )$, where $\theta \in \Omega$, ${\hat{\theta }}_n$ be a consistent Sequence of solutions of MLE equation $\frac{\partial l(\theta )}{\partial \theta }=0$, and $0<I({\theta }_0)<\infty$, then $\sqrt{n}({\hat{\theta }}_n-{\theta }_0)\stackrel{D}{\to }N\left(0,\frac{1}{I({\theta }_0)}\right)$ where $I({\theta }_0)$ is the Fisher Information.

By the asymptotic normality, the MLE estimator is asymptotically efficient under R0 ~ R5 Regularity Conditions

Asymptotic Confidence Interval

By the asymptotic normality of MLE, $\sqrt{nI(\hat{\theta })}(\hat{\theta }-\theta )\stackrel{D}{\to }N(0,1)$ Thus, $100(1-\alpha )\%$ confidence interval of for $\theta$ is $\left(\hat{\theta }-z_{\alpha /2}\frac{1}{\sqrt{nI(\hat{\theta })}},\hat{\theta }+z_{\alpha /2}\frac{1}{\sqrt{nI(\hat{\theta })}}\right)$

Delta method for MLE Estimator

Under the R0 ~ R5 Regularity Conditions, let $g(x)$ be a continuous function and $g^{\prime }({\theta }_0)\ne 0$, then $\sqrt{n}(g({\hat{\theta }}_n)-g({\theta }_0))\stackrel{D}{\to }N\left(0,\frac{g^{\prime }({\theta }_0{)}^2}{I({\theta }_0)}\right)$

Facts

Under R0 and R1 regularity conditions, let ${\theta }_0$ be a true parameter, then $\forall \theta \ne {\theta }_0,\underset{n\to \infty }{\lim }{P}_{{\theta }_0}[L({\theta }_0)>L(\theta )]=1$