Generalized Maximum Likelihood Estimation

Definition

Assume that the observation $x$ has a Probability Measure $P_{\theta }$ which satisfy $d{P}_{\theta }(x)=f_{\theta }(x)d\mu (x)\iff P_{\theta }(A)={\int }_A{f}_{\theta }(x)d\mu (x)$ where $\mu (x)$ is a dominating measure for the class $\{P_{\theta }\}$.

The usual MLE $\hat{\theta }$ is obtained by maximizing the Likelihood Function $L(\theta )=f_{\theta }(x)$ with respect to $\theta$, given the observed data $x$.

Assume that our observation has a Probability Measure $P_F$ that depends on the unknown Distribution Function $F$. The class $\{P_{\theta }\}$ does not have any dominating measure, so we need a more general definition of MLE.

To overcome this difficulty, Kiefer and Wolfowitz suggested the concept of the generalized maximum likelihood estimator (GMLE). Let $\mathcal{P}=\{P_{\theta }\}$ be a class of probability measures. For $P_1,P_2\in \mathcal{P}$, define the Radon–Nikodym derivative of $P_1$ with respect to $P_1+P_2$ as $f(x;P_1,P_2)=\frac{d{P}_1(x)}{d(P_1+P_2)}$

The generalized maximum likelihood estimator satisfies the following condition $\forall P\in \mathcal{P},f(x;\hat{P},P)\ge f(x;P,\hat{P})$