Mathematical Statistics Note

Probability and Distributions

The Probability Set Function

Sigma-Algebra

Definition

A sigma-algebra $\mathcal{F}$ is a Family of Sets satisfying the following properties

• $\Omega \in \mathcal{F}\text{or}\mathcal{F}\ne \varnothing$
• $\forall A\in \mathcal{F},A^{\complement }\in \mathcal{F}$
• $\forall \{A_i{\}}_{i=1}^{\infty }\subset \mathcal{F},\bigcup _{i=1}^{\infty }{A}_i\in \mathcal{F}$

where $\Omega$ is a universal set

Intersection of Sigma-Fields

Consider a set of sigma-fields ${\mathcal{F}}_i$ on $\Omega$,

• ${\mathcal{F}}_1\cap {\mathcal{F}}_2$ is a sigma-field on $\Omega$.
• $\bigcup _{i=1}^{\infty }{\mathcal{F}}_i$ is a sigma-field on $\Omega$.
• $\bigcup ^{\infty }{\mathcal{F}}_i$ is a sigma-field on $\Omega$

Facts

If a set of subsets $\mathcal{A}$ of a universal set $\Omega$ is both Pi-System and Lambda-System, then $\mathcal{A}$ is a sigma-algebra on $\Omega$.

Link to original

Probability Measure

Definition

The Measure with normality $\mu (\Omega )=1$

Link to original

Inclusion-Exclusion Formula

Definition

$\left|{\bigcup }_{i=1}^n{A}_i\right|={\sum }_{i=1}^n|A_i|-{\sum }_{1\le i<j\le n}|A_i\cap A_j|+{\sum }_{1\le i<j<k\le n}|A_i\cap A_j\cap A_k|-\cdots +(-1{)}^{n+1}\left|A_1\cap \cdots \cap A_n\right|$

Link to original

Increasing and Decreasing Sequence of Events

Definition

$\underset{n\to \infty }{\lim }{C}_n=\{\begin{array}{ll}\bigcup _{n=1}^{\infty }{C}_n& \text{for increasing sequence}\\ \bigcap _{n=1}^{\infty }{C}_n& \text{for decreasing sequence}\end{array}$

Facts

Let $\{C_n\}$ be an increasing sequence of events, then $\underset{n\to \infty }{\lim }P(C_n)=P(\underset{n\to \infty }{\lim }{C}_n)=P(\bigcup _{n=1}^{\infty }{C}_n)$

Let $\{C_n\}$ be a decreasing sequence of events, then $\underset{n\to \infty }{\lim }P(C_n)=P(\underset{n\to \infty }{\lim }{C}_n)=P(\bigcap _{n=1}^{\infty }{C}_n)$

Link to original

Boole's Inequality

Definition

$P(\bigcup _{n=1}^{\infty }{C}_n)\le \sum _{n=1}^{\infty }P(C_n)$

Link to original

Conditional Probability

Definition

Facts

$P(C_1\cap C_2\cap C_3\cap \dots )=P(C_1)P(C_2|C_1)P(C_3|C_1\cap C_2)P(C_4|C_1\cap C_2\cap C_3)\dots$

Link to original

Bayes Theorem

Definition

Discrete Case

$P(C_j|C)=\frac{P(C|C_j)P(C_j)}{P(C)}=\frac{P(C|C_j)P(C_j)}{\sum _{i=1}^{k}P(C|C_i)P(C_i)},j=1,\dots ,k$ where $\sum _{i=1}^{k}P(C_i)P(C|C_i)=P(C)$ by Law of Total Probability

$P(C_j)$ is called a prior probability, and $P(C_j|C)$ is called a posterior probability

Continuous Case

$f(\theta |x;\gamma )=\frac{f(x|\theta )f(\theta ;\gamma )}{f(x)}=\frac{f(x|\theta )f(\theta ;\gamma )}{\int f(x|\theta )f(\theta ;\gamma )d\theta }\propto f(x|\theta )f(\theta ;\gamma )$ where $\theta$ is not a constant, but an unknown parameter follows a certain distribution with a parameter $\gamma$.

$f(\theta ;\gamma )$ is called a prior probability, $f(x|\theta )$ is called a likelihood, $f(x)$ is called an evidence or marginal likelihood, and $f(\theta |x;\gamma )$ is called a posterior probability

Parameter-Centric Notation

$p(\theta |x)=\frac{p(x|\theta )\pi (\theta )}{p(x)}$

Examples

Consider a random variable follows Binomial Distribution $X\sim B(n,\theta )$ and a prior distribution follows Beta Distribution $\pi (\theta ;\gamma )\sim \mathrm{Beta}(\alpha ,\beta )$ where $\gamma =(\alpha ,\beta )$.

The PDFs are defined as $p(x|\theta )=(\genfrac{}{}{0ex}{}{n}{x}){\theta }^x(1-\theta {)}^{n-x}$ and $\pi (\theta ;\gamma )=\frac{\Gamma (\alpha +\beta )}{\Gamma (\alpha )\Gamma (\beta )}{\theta }^{\alpha -1}(1-\theta {)}^{\beta -1}$. Then, by Bayes theorem, $p(\theta |x)\propto p(x|\theta )\pi (\theta ;\gamma )\propto {\theta }^{x+\alpha -1}(1-{\theta }^{n-x+\beta -1})\sim \mathrm{Beta}(x+\alpha ,n-x+\beta )$ Under Squared Error Loss, the Bayes Estimator is a mean of the posterior distribution. ${\hat{\delta }}_{\text{Bayes}}=\frac{x+\alpha }{n+\alpha +\beta }$

Link to original

Statistical Independence

Definition

Statistical Independence of Events

Suppose a Probability Space $(\Omega ,\mathcal{F},P)$. The events $(E_{\alpha }{)}_{\alpha \in I}\subset \mathcal{F}$ are independent if $P(\bigcap _{j=1}^n{E}_{{\alpha }_j})=\prod _{j=1}^{n}P(E_{{\alpha }_j}),\forall ({\alpha }_j{)}_{j=1}^n\subset I$

Statistical Independence of Two Random Variables

Rigorous

Suppose a Probability Space $(\Omega ,\mathcal{F},P)$. random variables $(X_{\alpha }:(\Omega ,\mathcal{F})\to (\mathbb{R},\mathcal{R}){)}_{\alpha \in I}$ are independent if $(E_{\alpha }:=X_{\alpha }^{-1}(B_{\alpha }){)}_{\alpha \in I}$ are independent, $\forall B_{\alpha }\in \mathcal{R}$

Casual

Two random variables $X,Y$ are independent if and only if $\forall x,\forall y,F_{X,Y}(x,y)=F_X(x)F_Y(y)$

Statistical Independence of Random Variables

Mutually Independent

A collection of random variables $X_1,X_2,\dots ,X_n$ are mutually independent if and only if $F_{\mathbf{X}}(\mathbf{x})={\prod }_{i=1}^n{F}_{X_i}(x_i),\forall x_i$ where $F_{X_i}(x_i)$ is marginal CDF of $X_i$

Or equivalently, a collection of random variables $X_1,X_2,\dots ,X_n$ are mutually independent if and only if $f_{\mathbf{X}}(\mathbf{x})={\prod }_{i=1}^n{f}_{X_i}(x_i),\forall x_i$ where $f_{X_i}(x_i)$ is marginal PDF of $X_i$

Pointwise Independent

A collection of random variables $X_1,X_2,\dots ,X_n$ are Pointwise independent if and only if $\forall i\ne j,F_{X_i,X_j}(x_i,x_j)=F_{X_i}(x_i)F_{X_j}(x_j),\forall x_i,x_j$ where $F_{X_i}(x_i)$ is marginal CDF of $X_i$ and $F_{X_i,X_j}(x_i,x_j)$ is joint pdf of $X_i$ and $X_j$.

Statistical Independence of Stochastic Processes

Facts

Random variables $X,Y$ are independent $\iff \forall x,y,\exists f,g\text{s.t.}{f}_{X,Y}(x,y)=g(x)h(y)$

where $g(x)$ is a function of $x$ only, and $h(y)$ is a function of $y$ only.

Random variables $X,Y$ are independent $\iff \forall x,\forall y,F_{X,Y}(x,y)=F_X(x)F_Y(y)$

Random variables $X,Y$ are independent $\iff \forall x,\forall y,P(a<X\le b,c<Y\le d)=P(a<X\le b)P(c<Y\le d)$

Random variables $X,Y$ are independent $\Rightarrow E[u(X)v(Y)]=E[u(X)]E[v(Y)]$

If Moment Generating Function $M(t_1,t_2)$ exists and Random variables $X,Y$ are independent $\iff M(t_1,t_2)=M(t_1,0)M(0,t_2)$

Mutually independence $\underset{\times }{\stackrel{\circ }{\rightleftharpoons }}$ pairwise independence

If $\mathbf{X}$ is mutually independent, then $E\left[{\prod }_{i=1}^n{g}_i(X_i)\right]={\prod }_{i=1}^{n}E[g_i(X_i)]$

If random variables $(X_{\alpha }{)}_{\alpha \in I}$ are independent, $f_{\alpha }:\mathbb{R}\to \mathbb{R}$ are Borel measurable, so are $(f_{\alpha }(X_{\alpha }))$

Let Random variables $X,Y$. $X$ and $Y$ are independent $\underset{\times }{\stackrel{\circ }{\rightleftharpoons }}\mathrm{Cov}(X,Y)=0$

If $X$ and $Y$ follow normal distributions, $X$ and $Y$ are independent $\iff \mathrm{Cov}(X,Y)=0$

Link to original

Random Variables

Random Variable

Definition

$X:\Omega \to \mathbb{R}\text{s.t.}\forall B\in \mathcal{R},\exists \omega \in \Omega ,\text{s.t.}{X}^{-1}(B)=\{\omega :X(\omega )\in B\}\in \mathcal{F}$

A random variable is a function $X:\Omega \to \mathbb{R}$ whose inverse function $X^{-1}(B)$ is $\mathcal{F}$-measurable for the two measurable spaces $(\Omega ,\mathcal{F})$ and $(\mathbb{R},\mathcal{R})$.

The inverse image of an arbitrary Borel set of Codomain $\mathbb{R}$ is an element of sigma field $\mathcal{F}$.

Notations

Consider a probability space $(\Omega ,\mathcal{F},\mathbb{P})$

• Outcomes: $H,T$
• Set of outcomes (Sample space): $\Omega =\{H,T\}$
• Events: $\varnothing ,\{H\},\{T\},\Omega$
• Set of events (Sigma-Field): $\mathcal{F}=\{\varnothing ,\{H\},\{T\},\{H,T\}\}$
• Probabilities: $\mathbb{P}:\mathcal{F}\to [0,1]$
• Random variable: $X:\Omega \to \mathbb{R}$

For a random variable $X$ on a Probability Space $(\Omega ,\mathcal{F},P)$

• $X\sim {\mu }_X$ if and only if the Distribution of $X$ is ${\mu }_X$
• $X\sim F_X$ if and only if the Distribution Function of $X$ is $F_X$

For a random variable $X$ on Probability Space $({\Omega }_X,{\mathcal{F}}_X,P_X)$ and another random variable $Y$ on Probability Space $({\Omega }_Y,{\mathcal{F}}_Y,P_Y)$

• $X\stackrel{d}{=}Y⟺\forall B\in \mathcal{R}:{\mu }_X(B)={\mu }_Y(B)$
• $X\stackrel{d}{=}Y⟺\forall k\in \mathbb{R}:F_X(k)=F_Y(k)$
• $X\stackrel{d}{=}Y⟺\forall k\in \mathbb{R}:P_X(X\le k)=P_Y(Y\le k)$

Link to original

Transclude of Density-Function

Distribution Function

Definition

$F_X(x):\mathbb{R}\to [0,1]:={\mu }_X((-\infty ,x])=P(X\le x)$

A distribution function $F_X$ is a function for the Random Variable on Probability Space $(\Omega ,\mathcal{F},P)$

Facts

$(\Omega ,\mathcal{F},P)\sim (\mathbb{R},\mathcal{R},{\mu }_X)$

Proof By definition, $F_X(x)={\mu }_X((-\infty ,x])$ So, defining $\forall x\in \mathbb{R},F_X(x)$ is the Equivalence Relation to defining $\mathcal{A}=\{(-\infty ,x]:x\in \mathbb{R}\},{\mu }_X:\mathcal{A}\to [0,1]$

Since $\mathcal{A}$ is a Pi-System, ${\mu }_X:\sigma (\mathcal{R})=\mathcal{R}\to [0,1]$ is uniquely determined by ${\mu }_X:\mathcal{A}\to [0,1]$ by Extension from Pi-System

Now, $(\mathbb{R},\mathcal{R},{\mu }_X)$ is a Measurable Space induced by $X$. Therefore, defining ${\mu }_X$ on $(\mathbb{R},\mathcal{R})$ is equivalent to the defining $P$ on $(\Omega ,\mathcal{F})$

Distribution function(CDF) $F:\mathbb{R}\to [0,1]$ has the following properties

• Monotonic increasing: $\forall a,b,\in \mathbb{R},a<b\Rightarrow F(a)\le F(b)$
• $\underset{x\to -\infty }{\lim }F(x)=0$
• $\underset{x\to \infty }{\lim }F(x)=1$
• Right-continuous: $\underset{x\to x_0^{+}}{\lim }F(x)=F(x_0)$

If a function $F:\mathbb{R}\to [0,1]$ satisfies the following properties, then $F$ is a distribution function(CDF) of some Random Variable $X$

• Monotonic increasing: $\forall a,b,\in \mathbb{R},a<b\Rightarrow F(a)\le F(b)$
• $\underset{x\to -\infty }{\lim }F(x)=0$
• $\underset{x\to \infty }{\lim }F(x)=1$
• Right-continuous: $\underset{x\to x_0^{+}}{\lim }F(x)=F(x_0)$

$\forall a,b\in \mathbb{R},a<b\Rightarrow (P(a<X\le b)=F_X(b)-F_X(a))$

Link to original

Transformation of Random Variable

Definition

$Y=g(X)$

A new Random Variable $Y$ can be defined by applying a function $g$ to the outcomes of a Random Variable $X$

Discrete

Transformation Technique

Suppose $g:X\to Y$ is Bijective function $P_Y(y)=P(Y=y)=p(g(X)=y)=P(X=g^{-1}(y))=P_X(g^{-1}(y))$

Continuous

Let $f_X(x)$ be the PDF of Random Variable $X$ with a support $S_X$, $g:X\to Y$ be a differentiable Bijective function, and $x=g^{-1}(y)$

Transformation Technique

$f_Y(y)=f_X(g^{-1}(y))\left|\frac{dx}{dy}\right|,y\in S_Y$ where $S_Y=\{y=g(x)|x\in S_X\}$

CDF Technique

If $g$ is increasing, then $F_Y(y)=P(Y\le y)=P(g(X)\le y)=P(X\le g^{-1}(y))=F_X(g^{-1}(y))$

If $g$ is decreasing, then $F_Y(y)=P(Y\le y)=P(g(X)\le y)=P(X\ge g^{-1}(y))=1-F_X(g^{-1}(y))$

Link to original

Expectation of a Random Variable

Expected Value

Definition

$E(X)={\int }_{\Omega }XdP={\int }_{\mathbb{R}}xd{\mu }_X={\int }_{\mathbb{R}}xd{F}_X$ The expected value of the Random Variable $X$ on Probability Space $(\Omega ,\mathcal{F},P)$

Continuous

$E(X)={\int }_{-\infty }^{\infty }x{f}_X(x)dx$ where $f_X(x)$ is a PDF of Random Variable $X$

The expected value of the Random Variable $X$ when $F_X$ satisfies absolute continuous over $\lambda$, ${\mu }_X<<\lambda$

Discrete

$E(X)={\sum }_{x}x{p}_X(x)$ where $p_X(x)$ is a PMF of Random Variable $X$

The expected value of the Random Variable $X$ when $F_X$ satisfies absolute continuous over $\#$, ${\mu }_X<<{\#}_X$ In other words, $F_X$ has at most countable jumps.

Expected Value of a Function

$E(g(x))={\int }_{\mathbb{R}}g(x)d{\mu }_X$

Continuous

$E(g(x))={\int }_{\mathbb{R}}g(x)f_X(x)dx$

Discrete

$E(g(x))=\sum _{x\in S_X}g(x)p_X(x)$

Properties

Linearity

Random Variables

If $\exists k_{i}E[X_i]$, then $E\left[{\sum }_{i=1}^n{k}_i{X}_i\right]={\sum }_{i=1}^n{k}_{i}E(X_i)$

Matrix of Random Variables

Let $W_1,W_2$ be a $m\times n$ matrices of random variables, $A_1,A_2$ be $k\times m$ matrices of constants, and $B$ a $n\times l$ matrix of constant. Then, $E[A_1{W}_1+A_2{W}_2]=A_{1}E[W_1]+A_{2}E[W_2]$

$E[A_1{W}_{1}B]=A_{1}E[W_1]B$

Notations

Expression	Discrete	Continuous
Expression for the event $2^{\Omega }$ and the probability $P$	${\sum }_{\omega \in \Omega }X(\omega )\cdot P(\omega )$	${\int }_{\Omega }X(\omega )\cdot dP(\omega )$
Expression for the measurable space $(\mathbb{R},\mathcal{R})$ and the distribution ${\mu }_X$	${\sum }_{x\in \mathbb{R}}x\cdot f(x)$	${\int }_{\mathbb{R}}f(x)\lambda (x)={\int }_{\mathbb{R}}\frac{d{\mu }_X}{d\lambda }\lambda (x)={\int }_{\mathbb{R}}d{\mu }_X={\int }_{\mathbb{R}}d{F}_X$

Link to original

Some Special Expectations

Moment

Definition

${\mu }_n^{\prime }=E(X^n)$ $n$-th moment

Facts

$\forall k\le m,(\exists E(X^m)\Rightarrow \exists E(X^k))$

Link to original

Central Moment

Definition

${\mu }_n=E[(X-\mu {)}^n]$ $n$-th central moment

A Moment of a random variable about its mean

Link to original

Moment Generating Function

Definition

Univariate

Let $X$ be a Random Variable $M_X(t)=E(e^{tX})=\sum _{i=0}^{\infty }\frac{E(X^i)}{i!}{t}^i$

Then, $M_X(t)$ is called the moment generating function of the $X$

Calculations of Moments

${\mu }_n^{\prime }=E(X^n)=M_X^{(n)}(0)$

Moment can be calculated using the mgf

Uniqueness of MGF

Let $X,Y$ be random variables with mgf $M_X(t),M_Y(t)$, respectively Then, $(\forall z\in \mathbb{R},F_X(z)=F_Y(z))\iff (\forall t\in (-h,h),M_X(t)=M_Y(t))$ where $h>0$

Let $X$ be a Random Variable If $\exists M_X(t),\forall t\text{s.t.}||t||\le t_0$, where $t_0\in {\mathbb{R}}^{+}$, then $M_X(t)$ determine the distribution uniquely.

Multivariate

Let $\mathbf{X}=(X_1,X_2,\dots ,X_n)$ be a Random Vector $M_{\mathbf{X}}(\mathbf{t})=E(e^{{\mathbf{t}}^{\prime }\mathbf{X}})$ where $\mathbf{t}=(t_1,\dots ,t_n)$ is a constant vector.

Calculations of Multivariate Moments

$\frac{{\partial }^{k+m}M(t_1,t_2)}{\partial t_1^k\partial t_2^m}{|}_{t_1=t_2=0}={\int }_{-\infty }^{\infty }{\int }_{-\infty }^{\infty }{x}^k{y}^{m}f(x,y)dxdy=E(X^k{Y}^m)$

Moment can be calculated using the mgf

Marginal MGF

$M_{\mathbf{X}}(t_1,0,\dots ,0)=M_{X_1}(t_1)$

Facts

The mgf may not exist.

Let $\mathbf{X}=(X_1,X_2,\dots ,X_n)$ be a Random Vector. If $M_{\mathbf{X}}(\mathbf{t})$ exists and can be expressed as $M_{\mathbf{X}}(\mathbf{t})=M_{\mathbf{X}}(t_1,\dots ,t_r,0,\dots ,0)M_{\mathbf{X}}(0,\dots ,0,t_{r+1},\dots ,t_n)$ then ${\mathbf{X}}_1=(X_1,\dots ,X_r{)}^{⊺}$ and ${\mathbf{X}}_2=(X_{r+1},\dots ,X_n{)}^{⊺}$ are statistically independent.

Let $\mathbf{X}=(X_1,X_2,\dots ,X_n)$ be a Random Vector, ${\mathbf{X}}_1=(X_1,\dots ,X_r{)}^{⊺}$, and ${\mathbf{X}}_2=(X_{r+1},\dots ,X_n{)}^{⊺}$, then ${\mathbf{X}}_1$ and ${\mathbf{X}}_2$ is independent $\iff \exists a,b\text{s.t.}{M}_{\mathbf{X}}(\mathbf{t})=a(t_1,\dots ,t_r)b(t_{r+1},\dots ,t_n)$ where $a$ and $b$ are functions.

Link to original

Characteristic Function

Definition

$\phi (t)=E(e^{itX})=E[\cos (tX)+i\sin (tX)]$

Link to original

Cumulant Generating Function

Definition

$K_X(t)=\ln M_X(t)=\ln E(e^{tX})$

Calculations of Cumulants

${\kappa }_n=K^{(n)}(0)$

the $n$-th cumulant can be calculated using cgf ${\kappa }_n=K^{(n)}(0)$

Link to original

Important Inequalities

Markov's Inequality

Definition

Let $X$ be a non-negative Random Variable, and $E(X)<\infty$, then $\forall a>0,P(X\ge a)\le \frac{E(X)}{a}$ where $\frac{E(X)}{a}$ serves as an upper bound for the probability $P(X\ge a)$

Extended Version for Non-negative Functions

Let $u(X)$ be a non-negative Function of a Random Variable $X$, and $E(X)<\infty$, then $\forall a>0,P[u(X)\ge a]\le \frac{E[u(X)]}{a}$ where $\frac{E[u(X)]}{a}$ serves as an upper bound for the probability $P[u(X)\ge a]$

Link to original

Chebyshev's Inequality

Definition

$\forall k\in \mathbb{R},P(|X-\mu |\ge k\sigma )\le \frac{1}{k^2}$

Link to original

Jensen's Inequality

Definition

Let $f$ be a Convex Function on an interval $I$, $X$ be a Random Variable with support $S\subset I$, and $E(X)<\infty$, then $f(E[X])\le E[f(X)]$

Facts

$\frac{1}{\frac{1}{n}\sum \frac{1}{a_i}}\le (\prod _{i=1}^n{a}_i{)}^{1/n}\le \frac{1}{n}\sum _{i=1}^n{a}_i$

By Jensen’s Inequality, the following relation is satisfied arithmetic mean $\le$ geometric mean $\le$ harmonic mean

Link to original

Multivariate Distribution

Random Vector

Definition

$\mathbf{X}:(\Omega ,\mathcal{F})\to ({\mathbb{R}}^d,{\mathcal{R}}^d),\mathbf{X}=(X_1,X_2,\dots ,X_d)$

Column vector whose components are random variables $X:(\Omega ,\mathcal{F})\to (\mathbb{R},\mathcal{R})$.

$\mathbf{X}(\omega )=(X_1,X_2,\dots ,X_d)(\omega )=(X_1(\omega ),X_2(\omega ),\dots ,X_d(\omega ))=(x_1,x_2,\dots ,x_d)$

Link to original

Multivariate Distribution

Definition

Joint CDF

The Distribution Function $F_{\mathbf{X}}:{\mathbb{R}}^n\to [0,1]$ of a Random Vector $\mathbf{X}=(X_1,\dots ,X_n{)}^{⊺}$ is defined as $F_{\mathbf{X}}(\mathbf{x})=P(\mathbf{X}\le \mathbf{x})=P({\mathbf{X}}_1\le x_1,\dots ,{\mathbf{X}}_n\le x_n)$ where $\mathbf{x}=(x_1,\dots ,x_n{)}^{⊺}$

Joint PDF

$p_{X_1,\dots ,X_n}(x_1,\dots ,x_n)=P(X_1=x_1,\dots ,X_n=x_n)$

The joint probability Distribution Function of $n$ discrete Random Vector $X_1,X_2,\dots ,X_n$

$f_{X_1,\dots ,X_n}(x_1,\dots ,x_n)=\frac{{\partial }^n{F}_{X_1,\dots ,X_n}(x_1,\dots ,x_n)}{\partial x_1\dots \partial x_n}$

The joint probability Distribution Function of $n$ continuous Random Vector $X_1,X_2,\dots ,X_n$

Expected Value of a Multivariate Function

Continuous

E[g(\mathbf{X})] = \idotsint\limits_{x_{n} \dots x_{1}} g(\mathbf{x})f(\mathbf{x})dx_{1} \dots \dots dx_{n}

Discrete

$E[g(\mathbf{X})]=\sum _{x_n}\cdots \sum _{x_1}g(\mathbf{x})f(\mathbf{x})$

Marginal Distribution of a Multivariate Function

Marginal CDF

$F_{X_1}(x_1)=\underset{x_2,\dots ,x_n\to \infty }{\lim }{F}_{\mathbf{X}}(\mathbf{x})$

Marginal PDF

f_{X_{1}}(x_{1}) = E[g(\mathbf{X})] = \idotsint\limits_{x_{n} \dots x_{1}} f(x_{2}, \dots, x_n)dx_{2} \dots \dots dx_{n}

Conditional Distribution of a Multivariate Function

$f_{2,\dots ,n|1}(x_2,\dots ,x_n|x_1)=\frac{f_{\mathbf{X}}(\mathbf{x})}{f_{X_1}(x_1)}$

$f_{1|2,\dots ,n}(x_1|x_2,\dots ,x_n)=\frac{f_{\mathbf{X}}(\mathbf{x})}{f_{X_2,\dots ,X_n}(x_2,\dots ,x_n)}$

Properties

Linearity

If $\exists E[g_1(X_1,X_2)],E[g_2(X_1,X_2)]$, then $E[k_1{g}_1(X_1,X_2)+k_2{g}_2(X_1,X_2)]=k_{1}E[g_1(X_1,X_2)]+k_{2}E[g_2(X_1,X_2)]$

Link to original

Transformation of Random Vector

Definition

$\mathbf{Y}=g(\mathbf{X})$

Discrete

Let $\mathbf{X}$ be a Random Vector with joint pmf $p_{\mathbf{X}}(\mathbf{x})$ and support $S$, and $g:\mathbf{X}\to \mathbf{Y}$ be a vector Bijective transformation $p_{\mathbf{Y}}(\mathbf{y})=P(\mathbf{Y}=\mathbf{y})=p(g(\mathbf{X})=\mathbf{y})=P(\mathbf{X}=g^{-1}(\mathbf{y}))=P_{\mathbf{X}}(g^{-1}(\mathbf{y}))$

Continuous

Let $\mathbf{X}$ be a Random Vector with joint pdf $f_{\mathbf{X}}(\mathbf{x})$ and support $S$, $g:\mathbf{X}\to \mathbf{Y}$ be a vector Bijective transformation, and $\mathbf{x}=g^{-1}(\mathbf{y})$

One-to-One Transformation Case

$f_{\mathbf{Y}}(\mathbf{y})=f_{\mathbf{X}}(g^{-1}(\mathbf{y}))\cdot |J|,\mathbf{y}\in S_{\mathbf{Y}}$ where $J=\frac{\partial \mathbf{x}}{\partial \mathbf{y}}=\left[\begin{array}{ccc}\frac{\partial x_1}{\partial y_1}& \cdots & \frac{\partial x_1}{\partial y_n}\\ ⋮& \ddots & ⋮\\ \frac{\partial x_m}{\partial y_1}& \cdots & \frac{\partial x_m}{\partial y_n}\end{array}\right]$ is a Jacobian Matrix, and $S_{\mathbf{Y}}=\{\mathbf{y}=g(\mathbf{x})|\mathbf{x}\in S_{\mathbf{X}}\}$

Many-to-One Transformation Case

Let $g:\mathbf{X}\to \mathbf{Y}$ be a many-to-one transformation, $\mathbf{x}=g^{-1}(\mathbf{y})$, $A_1,\dots ,A_k$ be a partition of a set $S_X={⨄}_{i=1}^k{A}_i$ whose each partition’s transformation result is one-to-one Then, $f_{\mathbf{Y}}(\mathbf{y})=\sum _{i=1}^k|J_i|f_{\mathbf{X}}(g_i^{-1}(\mathbf{y})),\mathbf{y}\in S_{\mathbf{Y}}$ where $J_i=\frac{\partial {\mathbf{x}}_i}{\partial \mathbf{y}}=\left[\begin{array}{ccc}\frac{\partial x_{1i}}{\partial y_1}& \cdots & \frac{\partial x_{1i}}{\partial y_n}\\ ⋮& \ddots & ⋮\\ \frac{\partial x_{mi}}{\partial y_1}& \cdots & \frac{\partial x_{mi}}{\partial y_n}\end{array}\right]$ is a Jacobian Matrix for each partition, and $S_{\mathbf{Y}}=\{\mathbf{y}=g(\mathbf{x})|\mathbf{x}\in S_{\mathbf{X}}\}$, and $\uplus$ is a union of the pairwise disjoint sets

CDF Technique

If $g$ is increasing, then $F_{\mathbf{Y}}(\mathbf{y})=P(\mathbf{Y}\le \mathbf{y})=P(g(\mathbf{X})\le \mathbf{y})=P(\mathbf{X}\le g^{-1}(\mathbf{y}))=F_{\mathbf{X}}(g^{-1}(\mathbf{y}))$

If $g$ is decreasing, then $F_{\mathbf{Y}}(\mathbf{y})=P(\mathbf{Y}\le \mathbf{y})=P(g(\mathbf{X})\le \mathbf{y})=P(\mathbf{X}\ge g^{-1}(\mathbf{y}))=1-F_{\mathbf{X}}(g^{-1}(\mathbf{y}))$

Link to original

Conditional Distribution

Conditional Distribution

Definition

$f_{Y|X}(y|x):=P(Y=y|X=x)=\frac{f_{X,Y}(x,y)}{f_X(x)}$ where $f_X(x)>0$

Link to original

Conditional Expectation

Definition

Conditional Expectation of a Random Variable

$E(X|Y=y)={\int }_{-\infty }^{\infty }x{f}_{X|Y}(x|y)dx$

Conditional Expectation of a Function

$E[g(x)|Y=y]={\int }_{-\infty }^{\infty }g(x)f_{X|Y}(x|y)dx$

Conditional Expectation with respect to a Sub-Sigma-Algebra

Consider a Probability Space $(\Omega ,\mathcal{F},P)$, a Random Variable $X:\Omega \to \mathbb{R}$, and a sub-Sigma-Algebra $\mathcal{G}\subset \mathcal{F}$. A conditional expectation of $X$ given $\mathcal{G}$, denoted as $E(X|\mathcal{G}):\Omega \to \mathbb{R}$ is a function which satisfies: $\forall G\in \mathcal{G},{\int }_{G}E(X|\mathcal{G})dP={\int }_{G}XdP$

Existence of Conditional Distribution

Define a measure ${\mu }^X(F)={\int }_{F}XdP$ and its restriction ${\mu }^X{|}_{\mathcal{G}}:={\mu }^X\circ g$ where $g:\mathcal{G}\to \mathcal{F},g(x)=x$, and $P{|}_{\mathcal{G}}:=P\circ g$. Then, ${\mu }^X{|}_{\mathcal{G}}\ll P{|}_{\mathcal{G}}$ (${\mu }^X{|}_{\mathcal{G}}$ is absolute continuous with respect to $P{|}_{\mathcal{G}}$) and there exists a Radon–Nikodym Theorem $\frac{d{\mu }^X{|}_{\mathcal{G}}}{dP{|}_{\mathcal{G}}}=:E(X|\mathcal{G})$ satisfying $\forall G\in \mathcal{G},{\mu }^X{|}_{\mathcal{G}}(G)={\int }_{G}E(X|\mathcal{G})d{P}_{\mathcal{G}}={\int }_{G}XdP={\mu }^X(G)$ and it is called the conditional expectation of $X$ given $\mathcal{G}$.

Conditional Expectation with respect to a Random Variable

Suppose a Probability Space $(\Omega ,\mathcal{F},P)$, and random variables $X:\Omega \to \mathbb{R}$ and $Y:\Omega \to \mathbb{R}$. The conditional expectation of $X$ given $Y$, denoted as $E(X|Y):\Omega \to \mathbb{R}$ is defined as $E(X|\sigma (Y))=E(X|\{Y^{-1}(B)|B\in \mathcal{R}\}\subset \mathcal{F})$ where $\sigma (Y)$ is the sigma-field generated by random variable $Y$

Facts

If $X$ is $\mathcal{G}$-measurable, then $E(XY|\mathcal{G})=XE(Y|\mathcal{G})$, where $\mathcal{G}\subset \mathcal{F}$ is sub-Sigma-Field

Since $X$ is $\mathcal{G}$-measurable, $\exists$ $\mathcal{G}$-measurable function $h$ s.t. $X=h(\mathcal{G})$ By the property of conditional expectation, $E(XY|\mathcal{G})=E(h(\mathcal{G})Y|\mathcal{G})=h(\mathcal{G})E(Y|G)=XE(Y|\mathcal{G})$.

Link to original

Conditional Variance

Definition

$\mathrm{Var}(Y|X)=E[(Y-E(Y|X){)}^2|X]=E[Y^2|X]-E[Y|X{]}^2$

Link to original

Law of Total Expectation

Definition

$E(X)=E_Y[E_X(X|Y)]$

Proof

By definition of conditional expectation $\forall G\in \mathcal{G},{\int }_{G}E(X|\mathcal{G})dP={\int }_{G}XdP$ Since $\mathcal{G}$ is sub-Sigma-Field, $\Omega \in \mathcal{G}$ and ${\int }_{\Omega }E(X|\mathcal{G})dP={\int }_{\Omega }XdP$ hold.

Then, By definition of Expected Value $E(X)={\int }_{\Omega }XdP$ $E(E(X|\mathcal{G}))={\int }_{\Omega }E(X|\mathcal{G})dP={\int }_{\Omega }XdP=E(X)$

Link to original

Law of Total Variance

Definition

$\mathrm{Var}[E(X|Y)]\le \mathrm{Var}(X)=\mathrm{Var}[E(X|Y)]+E[\mathrm{Var}(X|Y)]$

Link to original

The Correlation Coefficient

Covariance

Definition

$\mathrm{Cov}(X,Y)=E[(X-{\mu }_X)(Y-{\mu }_Y)]$

Properties

Covariance with Itself

$\mathrm{Cov}(X,X)=\mathrm{Var}(X)$

Covariance of Linear Combinations

$\mathrm{Cov}(\sum _{i=1}^n{a}_i{X}_i,\sum _{i=1}^n{b}_i{Y}_i)=\sum _{i=1}^n\sum _{j=1}^m{a}_i{b}_j\mathrm{Cov}(X_i,Y_j)$

$\mathrm{Var}(\sum _{i=1}^n{a}_i{X}_i)=\mathrm{Cov}(\sum _{i=1}^n{a}_i{X}_i,\sum _{i=1}^n{a}_i{X}_i)=\sum _{i=1}^n{a}_i^2\mathrm{Var}(X_i)+2\sum _{i,j:i<j}{a}_i{a}_j\mathrm{Cov}(X_i,X_j)$

Link to original

Pearson Correlation Coefficient

Definition

${\rho }_{X,Y}=\frac{\mathrm{cov}(X,Y)}{{\sigma }_X{\sigma }_Y}=\frac{E[(X-{\mu }_X)(Y-{\mu }_Y)]}{{\sigma }_X{\sigma }_Y}$

Facts

Let $E(Y|X)$ is linear in $X$, that is $E(Y|X)=a+bX$, then the Conditional Expectation and Conditional Variance will be $E(Y|X)={\mu }_2+\rho \frac{{\sigma }_2}{{\sigma }_1}(X-{\mu }_1)$ $E[Var(Y|X)]={\sigma }_2^2(1-{\rho }^2)$

Link to original

Independent Random Variable

Independent and Identically Distributed Random Variable

Definition

A Random Vector is independent and identically distributed if each Random Variable has the same Distribution and are mutually independent.

Facts

If $\mathbf{X}$ is i.i.d. Random Vector, then $E(\mathbf{X})=(E(X_1),\dots ,E(X_n){)}^{⊺}$

Link to original

Extension to Several Random Variables

Covariance Matrix

Definition

Variance-Covariance Matrix

Let $\mathbf{X}=(X_1,X_2,\dots ,X_n)$ be an $n$ dimensional Random Vector with finite variance $Var(X_i)<\infty$, and $\mathbf{\mu }=E[\mathbf{X}]$, then the variance-covariance matrix of $\mathbf{X}$ is $\mathrm{Var}(\mathbf{X})=\mathrm{Cov}(\mathbf{X},\mathbf{X})=E[(\mathbf{X}-\mathbf{\mu })(\mathbf{X}-\mathbf{\mu }{)}^{⊺}]=E(X{X}^{⊺})-E(X)E(X{)}^{⊺}$

Cross-Covariance Matrix

Let $\mathbf{X}=(X_1,X_2,\dots ,X_n),\mathbf{Y}=(Y_1,Y_2,\dots ,Y_n)$ be $n$ dimensional random vectors with finite variance $Var(X_i)<\infty ,Var(Y_i)<\infty$, and ${\mathbf{\mu }}_{\mathbf{X}}=E[\mathbf{X}],{\mathbf{\mu }}_{\mathbf{Y}}=E[\mathbf{Y}]$, then the cross-covariance matrix of $\mathbf{X},\mathbf{Y}$ is $\mathrm{Cov}(\mathbf{X},\mathbf{Y})=E[(\mathbf{X}-{\mathbf{\mu }}_{\mathbf{X}})(\mathbf{Y}-{\mathbf{\mu }}_{\mathbf{Y}}{)}^{⊺}]=[\mathrm{Cov}(X_i,Y_i)]=E(X{Y}^{⊺})-{\mu }_X{\mu }_Y^{⊺}$

Properties

Let $\mathbf{X}$ be a Random Vector with finite variance $Var(X_i)<\infty$, and $A$ be a matrix of constants, then

$\mathrm{Var}(A\mathbf{X})=A\mathrm{Var}(\mathbf{X})A^{⊺}$

Let $X$ and $Y$ be random vectors. $\mathrm{Cov}(AX,BY)=A\mathrm{Cov}(X,Y)B^{⊺}$ where $A$ and $B$ are matrices of constants.

Facts

Every variance-covariance matrix is Positive Semi-Definite Matrix

Let $X$ be a Random Vector such that no element of $X$ is a linear combination of the remaining elements. $\nexists \mathbf{a},b\text{s.t.}\mathbf{a}X=b$ Then, the Variance-Covariance Matrix of the random vector is Positive-Definite Matrix. $\mathrm{Var}(X)\succ 0$

Link to original

Some Special Distributions

Uniform Distribution

Definition

$X\sim U(a,b)$ where $a$ and $b$ are the minimum and maximum values.

Properties

PDF

$f(x)=\{\begin{array}{ll}\frac{1}{b-a}& x\in [a,b]\\ 0& \text{otherwise}\end{array}$

CDF

$F(x)=\{\begin{array}{ll}0& x<a\\ \frac{x-a}{b-a}& x\in [a,b]\\ 1& x>b\end{array}$

Mean

$E(X)=\frac{a+b}{2}$

Variance

$\mathrm{Var}(X)=\frac{(b-a{)}^2}{12}$

Link to original

The Binomial and Related Distributions

Bernoulli Distribution

Definition

$X\sim \mathrm{Ber}(p)=B(1,p)$ where $p\in [0,1]$ is a probability of success

number of success in a single trial with success probability $p$

Bernoulli Process

The i.i.d. Random Vector $X_1,\dots ,X_n$ with Bernoulli distribution

Properties

PDF

$f(x)=p^x(1-p{)}^{1-x},x\in \{0,1\}$

Mean

$E(X)=p$

Variance

$\mathrm{Var}(X)=p(1-p)$

Link to original

Binomial Distribution

Definition

$X\sim B(n,p)$ where $n$ is the length of bernoulli process, and $p\in [0,1]$ is a probability of success

The number of successes in length $n$ bernoulli process with success probability $p$

Properties

PDF

$f(x)=\left(\genfrac{}{}{0ex}{}{n}{x}\right)p^x(1-p{)}^{n-x},x\in \{0,1,\dots ,n\}$

Mean

$E(X)=np$

Variance

$\mathrm{Var}(X)=np(1-p)$

MGF

$M_X(t)=[(1-p)+p{e}^t{]}^n$

Facts

Let $X_i\sim B(n_i,p)$ be independent random variables following binomial distribution, then $Y=\sum _{i=1}^n{X}_i\sim B(\sum _{i=1}^n{n}_i,p)$

Link to original

Negative Binomial Distribution

Definition

$X\sim NB(r,p)$ where $r>0$ is a number of successes, and $p$ is a probability in each experiment

The number of failures in a bernoulli process before the $r$-th success

Properties

PDF

$f(y)=\left(\genfrac{}{}{0ex}{}{y+r-1}{y}\right)p^r(1-p{)}^y,x\in N_0$

Mean

$E(X)=\frac{r(1-p)}{p}$

Variance

$\mathrm{Var}(X)=\frac{r(1-p)}{p^2}$

MGF

$M_Y(t)=p^r[1-(1-p)e^t{]}^{-r}={\left(\frac{p}{1-(1-p)e^t}\right)}^r$

Link to original

Geometric Distribution

Definition

$Y\sim \mathrm{Geom}(p)=NB(1,p)$ where $p$ is a probability in each experiment

The number of failures before the 1st success

Properties

PDF

$f(y)=p(1-p{)}^y,y\in {\mathbb{N}}_0$

Mean

$E(X)=\frac{1-p}{p}$

Variance

$\mathrm{Var}(X)=\frac{1-p}{p^2}$

Link to original

Trinomial Distribution

Definition

$X\sim \mathrm{Tri}(n,p_1,p_2)$ where $n$ is the number of trials, $p_1$ is a probability of category $1$, and $p_2$ is a probability of category $2$

Distribution that expands the outcomes of a binomial distribution to three

Properties

PDF

$f(x,y)=\frac{n!}{x!y!(n-x-y)!}{p}_1^x{p}_2^y(1-p_1-p_2{)}^{n-x-y}$

MGF

$M_{X,Y}(t_1,t_2)=(p_1{e}^{t_1}+p_2{e}^{t_2}+p_3{)}^n$

Marginal PDF

$f_X(x)=\left(\genfrac{}{}{0ex}{}{n}{x}\right)p_1^x(1-p_1{)}^{n-x}\sim B(n,p_1)$

$f_Y(y)\sim B(n,p_2)$

Conditional PDF

$f_{Y|X}(y|x)=(\genfrac{}{}{0ex}{}{n-x}{y}){\left(\frac{p_2}{1-p_1}\right)}^y{\left(1-\frac{p_2}{1-p_1}\right)}^{n-x-y}\sim B\left(n-x,\frac{p_2}{1-p_1}\right)$

$f_{X|Y}(x|y)\sim B\left(n-y,\frac{p_1}{1-p_2}\right)$

Link to original

Multinomial Distribution

Definition

$X\sim \mathrm{Mult}(n,p_1,p_2,\dots ,p_{k-1})$ where $n$ is the number of trials, $p_i$ is a probability of category $i$

Distribution that describes the probability of observing a specific combination of outcomes

Properties

PDF

$f(x_1,x_2,\dots ,x_{k-1})=n!{\prod }_{i=1}^k\frac{p_i^{x_i}}{x_i!}$ where $x_k=n-\sum _{i=1}^{k-1}{x}_i$, $p_k=1-\sum _{i=1}^{k-1}{p}_i$, and $0\le \sum _{i=1}^{k-1}{x}_i\le n$

MGF

$M(t_1,t_2,\dots ,t_{k-1})=({\sum }_{i=1}^{k-1}{p}_i{e}^{t_i}+p_k{)}^n$

Marginal PDF

Each one-variable marginal pdf is Binomial Distribution, each two-variables marginal pdf is Trinomial Distribution, and so on.

Link to original

Hypergeometric Distribution

Definition

$X\sim \mathrm{Hypergeom}(N,K,n)$ where $N$ is the population size, $K$ is the number of success, and $n$ is the number of trials,

Distribution that describes the probability of $k$ successes in $n$ draws, without replacement, from a population $N$ that contains $K$ successes.

Draws without replacement version of Binomial Distribution

Properties

PDF

$f(x)=\frac{\left(\genfrac{}{}{0ex}{}{K}{k}\right)\left(\genfrac{}{}{0ex}{}{N-K}{n-k}\right)}{\left(\genfrac{}{}{0ex}{}{N}{n}\right)},x\in \{0,1,\dots ,n\}$

Mean

$E(X)=n\frac{K}{N}$

Variance

$\mathrm{Var}(X)=n\frac{K}{N}\frac{N-K}{N}\frac{N-n}{N-1}$

Link to original

Poisson Distribution

Poission Distribution

Definition

$X\sim \mathrm{Pois}(\lambda )$ where $\lambda$ is the average number of occurrences in a fixed interval of time

The number of occurrences in a fixed interval of time with mean $\lambda$

Properties

PDF

$f(x)=\frac{e^{-\lambda }{\lambda }^x}{x!}\in [0,1]$ where $x\in {\mathbb{N}}_0$

Mean

$E(X)=\lambda$

Variance

$\mathrm{Var}(X)=\lambda$

MGF

$M(t)=\exp [m(e^t-1)]$

Summation

Let $X_i\sim \mathrm{Pois}({\lambda }_i)$, and $X_i$‘s are independent, then $\sum _{i=1}^n{X}_i\sim \mathrm{Pois}\left(\sum _{i=1}^n{\lambda }_i\right)$

Link to original

The Gamma, Chi-Squared, and Beta Distribution

Gamma Function

Definition

$\Gamma (\alpha ):={\int }_0^{\infty }{y}^{\alpha -1}{e}^{-y}dy$ where $\alpha \in {\mathbb{R}}^{+}$

$\Gamma (s):={\int }_0^1(-\log u{)}^{s-1}du$ where $s\in {\mathbb{R}}^{+}$

Gamma function is used as an extension of the factorial function

Properties

$\forall 1<\alpha \in \mathbb{R},\Gamma (\alpha )=(\alpha -1)\Gamma (\alpha -1)$

$\forall \alpha \in \mathbb{N},\Gamma (\alpha )=(\alpha -1)!$

$\Gamma \left(\frac{1}{2}\right)=\sqrt{\pi }$

Link to original

Gamma Distribution

Definition

$X\sim \Gamma (\alpha ,\beta )=\Gamma (k,\theta )$ where $\alpha =k\in {\mathbb{R}}^{+}$ is the shape parameter, and $\beta =\frac{1}{\theta }\in {\mathbb{R}}^{+}$ is the scale parameter

Distribution that models the waiting time until occurring $\alpha$th events in a Poisson process with mean $\theta =\frac{1}{\beta }$

Properties

PDF

$f(x)=\frac{x^{k-1}{e}^{-\frac{x}{\theta }}}{\Gamma (k){\theta }^k}=\frac{{\beta }^{\alpha }{x}^{\alpha -1}{e}^{-\beta x}}{\Gamma (\alpha )},x\in {\mathbb{R}}^{+}$ where $\Gamma (\alpha )$ is the Gamma Function

Mean

$E(X)=k\theta =\frac{\alpha }{\beta }$

Variance

$\mathrm{Var}(X)=k{\theta }^2=\frac{\alpha }{{\beta }^2}$

MGF

$M(t)=(1-\theta t{)}^{-k}={\left(1-\frac{t}{\beta }\right)}^{-\alpha }$ where $t<\frac{1}{\theta }$ and $t<\beta$

Sum of independent Gamma distributions

Let $X_i\sim \Gamma ({\alpha }_i,\beta )$‘s are independent gamma distributions $\sum _{i=1}^n{X}_i\sim \Gamma \left(\sum _{i=1}^n{\alpha }_i,\beta \right)$

Scaling

Let $X\sim \Gamma (k,\theta )$, then, $\forall c>0,cX\sim \Gamma (k,c\theta )$

Let $X\sim \Gamma (\alpha ,\beta )$, then, $\forall c>0,cX\sim \Gamma (\alpha ,\frac{\beta }{c})$

Relationship with Poisson Distribution

Let $W$ be the time until the $k$-th event in a poisson process with rate $\lambda$, then, the cdf and pdf of $W$ is $G(w)={\int }_0^w\frac{{\lambda }^k{y}^{k-1}{e}^{-\lambda y}}{\Gamma (k)}dy,w\in {\mathbb{R}}^{+}$ $g(w)=\frac{{\lambda }^k{w}^{k-1}{e}^{-\lambda w}}{\Gamma (k)}\sim \Gamma \left(k,\frac{1}{\lambda }\right),w\in {\mathbb{R}}^{+}$

Link to original

Exponential Distribution

Definition

$X\sim \mathrm{Exp}(\lambda )={\Gamma }_{\alpha ,\beta }\left(1,\frac{1}{\lambda }\right)={\Gamma }_{k,\theta }(1,\lambda )$ where $\lambda \in {\mathbb{R}}^{+}$ is the average number of events per fixed unit of time

Time interval between events in a Poisson process with mean $\lambda$

Properties

PDF

$f(x)=\lambda e^{-\lambda x}\in {\mathbb{R}}_0^{+}$

CDF

$F(x)=1-e^{-\lambda x}$

Mean

$E(X)=\frac{1}{\lambda }$

Variance

$\mathrm{Var}(X)=\frac{1}{{\lambda }^2}$

Memorylessness Property

$\forall x,t\ge 0:P(T>s+t|T>s)=P(T>t)$

Link to original

Chi-squared Distribution

Definition

$X\sim {\chi }^2(k)=\Gamma \left(\frac{k}{2},2\right)$ where $k\in \mathbb{N}$ is the degrees of freedom

squared sum of independent standard normal distributions

Properties

PDF

$f(x)=\frac{1}{\Gamma (\frac{k}{2})2^{\frac{k}{2}}}{x}^{\frac{k}{2}-1}{e}^{-\frac{x}{2}}\in {\mathbb{R}}_0^{+}$

Mean

$E(X)=k$

Variance

$\mathrm{Var}(X)=2k$

MGF

$M(t)=(1-2t{)}^{-k/2}$

Additivity

Let $X_i\sim {\chi }^2(k_i)$‘s are independent chi-squared distributions $\sum _{i=1}^n{X}_i\sim {\chi }^2\left(\sum _{i=1}^n{k}_i\right)$

Facts

Let $Q_1\sim {\chi }^2(r_1)$ and $Q_2\sim {\chi }^2(r_2)$, and $Q=Q_1-Q_2$ is independent of $Q_2$, then $Q\sim {\chi }^2(r_1-r_2)$

Let $Q=Q_1+Q_2+\cdots +Q_{k-1}+Q_k$, where $Q,Q_1,Q_2,\dots ,Q_k$ are quadratic forms in $\mathbf{x}$, where each element of the $\mathbf{x}$ is a Random Sample from $N(\mu ,{\sigma }^2)$ If $Q/{\sigma }^2\sim {\chi }^2(r),Q_1/{\sigma }^2\sim {\chi }^2(r_1),Q_2/{\sigma }^2\sim {\chi }^2(r_2),\dots ,Q_{k-1}/{\sigma }^2\sim {\chi }^2(r_{k-1})$, then

• $Q_1,Q_2,\dots ,Q_k$ are independent
• $Q_k/{\sigma }^2\sim {\chi }^2(r-r_1-r_2-\cdots -r_{k-1})$

Link to original

Beta Distribution

Definition

$X\sim \mathrm{Beta}(\alpha ,\beta )$ where $\alpha ,\beta \in {\mathbb{R}}^{+}$ are the shape parameters

Properties

PDF

$f(x)=\frac{\Gamma (\alpha +\beta )}{\Gamma (\alpha )\Gamma (\beta )}{x}^{\alpha -1}(1-x{)}^{\beta -1}\in (0,1)$

Mean

$E(X)=\frac{\alpha }{\alpha +\beta }$

Variance

$\mathrm{Var}(X)=\frac{\alpha \beta }{(\alpha +\beta +1)(\alpha +\beta {)}^2}$

Derivation from Gamma Distribution

Let $X_1\sim \Gamma (\alpha ,1),X_2\sim \Gamma (\beta ,1)$ are independent gamma distributions, then $\frac{X_1}{X_1+X_2}\sim \mathrm{Beta}(\alpha ,\beta )$

Link to original

Dirichlet Distribution

Definition

$X\sim \mathrm{Dir}(a_1,a_2,\dots ,a_K)$ where $K\in \mathbb{N}$ is the number of categories, and $a_i\in {\mathbb{R}}^{+}$‘s are the concentration parameters

A multivariate generalization of the Beta Distribution

Properties

PDF

$f(x)=\frac{\Gamma ({\sum }_{i=1}^K{\alpha }_i)}{{\prod }_{i=1}^K\Gamma ({\alpha }_i)}{\prod }_{i=1}^K{x}_i^{{\alpha }_i-1}$

Facts

If $K=1$, then the Dirichlet distribution is a beta distribution.

Link to original

The Normal Distribution

Normal Distribution

Definition

$X\sim N(\mu ,{\sigma }^2)$ where $\mu \in \mathbb{R}$ is the location parameter(mean), and ${\sigma }^2\in {\mathbb{R}}_0^{+}$ is the scale parameter(variance)

Standard Normal Distribution

$X\sim N(0,1)$ $f(x)=\frac{1}{\sqrt{2\pi }}\exp \left(-\frac{x^2}{2}\right)\in \mathbb{R}$

Properties

PDF

$f(x)=\frac{1}{\sqrt{2\pi {\sigma }^2}}\exp \left(-\frac{1}{2}(\frac{x-\mu }{\sigma }{)}^2\right)\in \mathbb{R}$

Mean

$E(X)=\mu$

Variance

$\mathrm{Var}(X)={\sigma }^2$

MGF

$M_X(t)=\exp \left(\mu t+\frac{{\sigma }^2{t}^2}{2}\right)$

Higher Order Moments

$E(Z^k)=\{\begin{array}{l}E(Z^{2k})=\frac{(2k)!}{2^{k}k!}\\ E(Z^{2k+1})=0\end{array}$

$E(X^k)=\sum _{j=0}^k\left(\genfrac{}{}{0ex}{}{k}{j}\right){\sigma }^{j}E(Z^j){\mu }^{k-j}$

Sum of Normally Distributed Random Variables

Let $X_i\sim N({\mu }_i,{\sigma }_i^2)$ be independent random variables following normal distribution, then $\sum _{i=1}^n{a}_i{X}_i\sim N\left(\sum _{i=1}^n{a}_i{\mu }_i,{\sum }_{i=1}^n{a}_i^2{\sigma }_i^2\right)$

Relationship with Chi-squared Distribution

Let $Z\sim N(0,1)$ be a standard normal distribution, then $Z^2\sim {\chi }^2(1)$

Facts

Let $X\sim N(0,{\sigma }^2)$, then $\mathrm{Var}(X^2)=2{\sigma }^4$

Link to original

The Multivariate Normal Distribution

Multivariate Normal Distribution

Definition

$\mathbf{X}\sim N_n(\mathbf{\mu },\Sigma )$ where $n$ is the number of dimensions, $\mathbf{\mu }$ is the vector of location parameters, and $\mathbf{\Sigma }$ is the vector of scale parameters

Standard Multivariate Normal Distribution

$\mathbf{X}\sim N_n(0,I_n)$

PDF

$f(\mathbf{z})=(2\pi {)}^{-\frac{n}{2}}\exp \left(-\frac{1}{2}{\mathbf{z}}^{⊺}\mathbf{z}\right)$

MGF

$M_{\mathbf{z}}(\mathbf{t})=\exp \left(\frac{1}{2}{\mathbf{t}}^{⊺}\mathbf{t}\right)$

Properties

PDF

$f(\mathbf{x})=(2\pi {)}^{-\frac{n}{2}}|\Sigma {|}^{-\frac{1}{2}}\exp \left(-\frac{1}{2}(\mathbf{x}-\mathbf{\mu }{)}^{⊺}{\Sigma }^{-1}(\mathbf{x}-\mathbf{\mu })\right)$

Mean

$E(\mathbf{X})=\mu$

Variance

$\mathrm{Var}(\mathbf{X})=\Sigma$

MGF

$M_{\mathbf{x}}(\mathbf{t})=\exp \left({\mathbf{t}}^{⊺}\mathbf{\mu }+\frac{1}{2}{\mathbf{t}}^{⊺}\Sigma \mathbf{t}\right)$

Affine Transformation

Let $\mathbf{X}\sim N_n(\mathbf{\mu },\Sigma )$ be a Random Variable following multivariate normal distribution, $A$ be a $m\times n$ matrix, and $\mathbf{b}$ be a $m$ dimensional vector, then $A\mathbf{X}+\mathbf{b}\sim N_m(A\mathbf{\mu }+\mathbf{b},A\Sigma A^{⊺})$

Relationship with Chi-squared Distribution

Suppose $\mathbf{X}\sim N_n(\mathbf{\mu },\Sigma )$ be a Random Variable following multivariate normal distribution, then ${\Sigma }^{-\frac{1}{2}}(\mathbf{X}-\mathbf{\mu })\sim N_n(0,I_n)$ $(\mathbf{X}-\mathbf{\mu }{)}^{⊺}{\Sigma }^{-1}(\mathbf{X}-\mathbf{\mu })\sim {\chi }^2(n)$

Facts

Let $\mathbf{X}\sim N_n(\mathbf{\mu },\Sigma )$, $C$ be an $m\times n$, and $\mathbf{d}$ be a $m$-dimensional vector, then $C\mathbf{X}+\mathbf{D}\sim N_m(C\mu +\mathbf{d},C\Sigma C^{⊺})$

Let $\mathbf{Y}\sim N_n(\mathbf{\mu },\Sigma )$, $\mathbf{Y}=(Y_1,Y_2{)}^{⊺}$, $\mathbf{\mu }=({\mu }_1,{\mu }_2{)}^{⊺}$, $\mathbf{\Sigma }=\left(\begin{array}{cc}{\Sigma }_{11}& {\Sigma }_{12}\\ {\Sigma }_{21}& {\Sigma }_{22}\end{array}\right)$, where $Y_1$ is $p\times 1$ and $Y_2$ is $(n-p)\times 1$ vectors. Then, $Y_1$ and $Y_2$ are independent $\iff {\Sigma }_{12}=O_{p\times (n-p)}$

Let $\mathbf{Y}\sim N_n(\mathbf{\mu },\Sigma )$, $\mathbf{u}=A\mathbf{Y}$, and $\mathbf{v}=B\mathbf{Y}$, where $A$ is $m\times n$, $B$ is $l\times n$ matrices. $\mathbf{u}$ and $\mathbf{v}$ are independent $\iff \mathrm{Cov}(\mathbf{u},\mathbf{v})=A\Sigma B^{⊺}=O_{m\times l}$

Link to original

t-and F-Distribution

Student's t-Distribution

Definition

Let $W\sim N(0,1)$ be a standard normal distribution, $V\sim {\chi }^2(r)$ be a Chi-squared Distribution, and $W,V$ be independent, then $\frac{W}{\sqrt{V/r}}\sim t(r)$ where $r\in {\mathbb{R}}^{+}$ is the degrees of freedom

Properties

PDF

$f_r(t)=\frac{\Gamma (\frac{r+1}{2})}{\sqrt{\pi r}\Gamma (\frac{r}{2})}{\left(1+\frac{t^2}{r}\right)}^{-(r+1)/2}\in \mathbb{R}$

Mean

$E(X)=0$

Variance

$\mathrm{Var}(X)=\{\begin{array}{ll}\frac{r}{r-2}& r>2\\ \infty & 1<r\le 1\end{array}$

Link to original

F-Distribution

Definition

Let $U\sim {\chi }^2(r_1),V\sim {\chi }^2(r_2)$ be independent random variables following Chi-squared distributions, then $\frac{U/r_1}{V/r_2}\sim F(r_1,r_2)$ where $r_1,r_2\in \mathbb{N}$ are the degrees of freedoms

Properties

PDF

{\Gamma(\frac{r_{1} + r_{2}}{2}) (\frac{r_{1}}{r_{2}})^{\frac{r_{1}}{2}} x^{\frac{r_{1}}{2}-1}} {\Gamma(\frac{r_{1}}{2}) \Gamma(\frac{r_{2}}{2}) (\frac{r_{1}}{r_{2}}x+1)^{(r_{1} + r_{2})/2}}$$ ## Mean $$E(X) = \frac{r_{2}}{r_{2} - 2},\quad r_{2}>2$$ ## Variance $$\operatorname{Var}(X) = \frac{2(r_{1}+r_{2}-2)}{r_{1} (r_{2}-2)^{2}(r_{2}-4)},\quad r_{2}>4$$Link to original

Student's Theorem

Definition

Let $X_i\sim N(\mu ,{\sigma }^2)$ be i.i.d. Random Vector following Normal Distribution, $\overline{X}:=\frac{1}{n}\sum _{i=1}^n{X}_i$, and $s^2:=\frac{1}{n-1}\sum _{i=1}^n(X_i-\overline{X}{)}^2$, then

• $\overline{X}\sim N(\mu ,\frac{{\sigma }^2}{n})$
• $\overline{X}$ and $s^2$ are independent
• $\frac{1}{{\sigma }^2}\sum _{i=1}^n(X_i-\overline{X}{)}^2=\frac{(n-1)s^2}{{\sigma }^2}\sim {\chi }^2(n-1)$
• $\frac{\overline{X}-\mu }{s/\sqrt{n}}\sim t(n-1)$

Link to original

Consistency, and Limiting Distributions

Convergence in Probability

Convergence in Probability

Definition

$(\forall ϵ>0,\underset{n\to \infty }{\lim }P(|X_n-X|\ge ϵ)=0)\iff (X_n\stackrel{P}{\to }X)$

Let $\{X_n\}$ be a Sequence of random variables and $X$ be a Random Variable, then $X_n$ converges in probability to $X$ if $\forall ϵ>0,\underset{n\to \infty }{\lim }P(|X_n-X|\ge ϵ)=0$, and denoted by $X_n\stackrel{P}{\to }X$

Facts

If the Sequence of random variables $\{X_n\}$ converges in probability to $X$, then it also Almost Surely converge to $X$.

$(X_n\stackrel{P}{\to }X)\Rightarrow (X_n\stackrel{D}{\to }X)$

Convergence in Probability implies convergence in distribution

Link to original

Convergence in probability has Linearity

• Additivity $({\overline{X}}_n\stackrel{P}{\to }X)\wedge ({\overline{Y}}_n\stackrel{P}{\to }Y)\Rightarrow X_n+Y_n\stackrel{P}{\to }X+Y$
• Homogeneity $\forall a\in \mathbb{R},X_n\stackrel{P}{\to }X\Rightarrow a{X}_n\stackrel{P}{\to }aX$

Continuous Mapping Theorem

Definition

Continuous functions preserve convergences (in probability, Almost Surely, or in distribution) of a Sequence of random variables to limits.

Consider a Sequence $\{X_n\}$ of random variables defined on same Probability Space, and a Continuous Function $g$ on the space. Then,

• Convergence in Probability to a Constant: $X_n\stackrel{P}{\to }a\in \mathbb{R}⟹g(X_n)\stackrel{P}{\to }g(a)$
• Convergence in Probability: $X_n\stackrel{P}{\to }X⟹g(X_n)\stackrel{P}{\to }g(X)$
• Almost Sure Convergence: $X_n\stackrel{a.s.}{\to }X⟹g(X_n)\stackrel{a.s.}{\to }g(X)$
• Convergence in Distribution: $X_n\stackrel{D}{\to }X⟹g(X_n)\stackrel{D}{\to }g(X)$

Link to original

Link to original

Consistency

Definition

${\hat{\mathbf{\theta }}}_n\stackrel{p}{\to }\mathbf{\theta }$

A Statistic ${\hat{\mathbf{\theta }}}_n$ is called consistent estimator of $\mathbf{\theta }$ if $\hat{\mathbf{\theta }}$ converges in probability to $\mathbf{\theta }$

Link to original

Converge in Distribution

Convergence in Distribution

Definition

$(\forall x\in C(F_X),\underset{n\to \infty }{\lim }{F}_{X_n}(x)=F_X(x))\Rightarrow (X_n\stackrel{D}{\to }X)$

Let $\{X_n\}$ be a Sequence of random variables with CDF $F_{X_n}$, $X$ be a Random Variable with cdf $F_X$, and $C(F_X)$ be a set of every point at which $F_X$ is continuous, then $X_n$ converges in distribution to $X$ if $\forall x\in C(F_X),\underset{n\to \infty }{\lim }{F}_{X_n}(x)=F_X(x)$, and denoted by $X_n\stackrel{D}{\to }X$. $X$ is called the limiting distribution of $X_n$ or asymptotic distribution of $X_n$

Facts

$(X_n\stackrel{P}{\to }X)\Rightarrow (X_n\stackrel{D}{\to }X)$

Convergence in Probability implies convergence in distribution

$(\forall c\in \mathbb{R},X_n\stackrel{P}{\to }c)\iff (X_n\stackrel{D}{\to }c)$

$(({\overline{X}}_n\stackrel{D}{\to }X)\wedge ({\overline{Y}}_n\stackrel{P}{\to }0))\Rightarrow (X_n+Y_n\stackrel{D}{\to }X)$

Continuous Mapping Theorem

Definition

Continuous functions preserve convergences (in probability, Almost Surely, or in distribution) of a Sequence of random variables to limits.

Consider a Sequence $\{X_n\}$ of random variables defined on same Probability Space, and a Continuous Function $g$ on the space. Then,

• Convergence in Probability to a Constant: $X_n\stackrel{P}{\to }a\in \mathbb{R}⟹g(X_n)\stackrel{P}{\to }g(a)$
• Convergence in Probability: $X_n\stackrel{P}{\to }X⟹g(X_n)\stackrel{P}{\to }g(X)$
• Almost Sure Convergence: $X_n\stackrel{a.s.}{\to }X⟹g(X_n)\stackrel{a.s.}{\to }g(X)$
• Convergence in Distribution: $X_n\stackrel{D}{\to }X⟹g(X_n)\stackrel{D}{\to }g(X)$

Link to original

Slutzky Theorem

Definition

Let $\{X_n\},\{Y_n\},\{Z_n\}$ be Sequence of random variables, $X$ be a Random Variable, $k_1,k_2$ be constants, then $((X_n\stackrel{D}{\to }X)\wedge (Y_n\stackrel{D}{\to }{k}_1)\wedge (Z_n\stackrel{D}{\to }{k}_2))\Rightarrow (Y_n+Z_n{X}_n\stackrel{D}{\to }{k}_1+k_{2}X)$

Link to original

Let $\{P_n{\}}_{n\in \mathbb{N}}$ be a sequence of distributions. Then, $D_{KL}(P_n||P)\to 0⟹D_{JS}(P_n,P)\to 0⟺\delta (P_n,P)\to 0⟹W(P_n,P)\to 0⟺P_n\stackrel{D}{\to }P$ The convergence of the KL-Divergence to zero implies that the JS-Divergence also converges to zero. The convergence of the JS-Divergence to zero is equivalent to the convergence of the Total Variation Distance to zero. The convergence of the Total Variation Distance to zero implies that the Wasserstein Distance also converges to zero. The convergence of the Wasserstein Distance to zero is equivalent to the Convergence in Distribution of the sequence.

Link to original

Link to original

Boundedness in Probability

Definition

$O_P(1)\iff \forall ϵ>0,\exists B\in {\mathbb{R}}^{+},\exists N\in \mathbb{N},\forall n\ge N,P(|X_n|\ge B)\le ϵ$

Facts

Let $\{X_n\}$ be a Random Vector, and $X$ be a Random Variable, then If $X_n\to X$, then $\{X_n\}$ is bounded in probability

Link to original

Big O Notation

Definition

Let $\{r_n\}\in (0,\infty )$ be the rate of convergence, and $\{x_n\}$ be a Sequence, then

$x_n=o(r_n)\iff \underset{n\to \infty }{\lim }\frac{x_n}{r_n}=0\iff \forall ϵ>0,\exists N\in \mathbb{N},\forall n\ge N,\left|\frac{x_n}{r_n}\right|<ϵ$

$x_n=O(R_n)\iff \underset{n\to \infty }{\mathrm{limsup}}\frac{x_n}{r_n}<\infty \iff \exists M\in {\mathbb{R}}^{+},\exists N\in \mathbb{N},\forall n\ge N,\left|\frac{x_n}{r_n}\right|\le M$

Link to original

Big O in Probability Notation

Definition

Extension of Big O Notation for Random Variable

Let $\{r_n\}\in (0,\infty )$ be the rate of convergence, and $\{X_n\}$ be a Sequence of random variables, then $X_n=o_P(r_n)\iff \underset{n\to \infty }{\lim }\frac{X_n}{r_n}\stackrel{P}{\to }0\iff \forall ϵ,M>0,\exists N\in \mathbb{N},\forall n\ge N,P(\left|\frac{X_n}{r_n}\right|\ge M)\le ϵ$

$X_n=O_P(r_n)\iff \forall ϵ>0,\exists M\in {\mathbb{R}}^{+},\exists N\in \mathbb{N},\forall n\ge N,P(\left|\frac{X_n}{r_n}\right|\ge M)\le ϵ$

Facts

If $X_n=O_P(1)$, then $\{X_n\}$ is called Boundedness in Probability

$X_n\stackrel{D}{\to }X\Rightarrow X_n=O_P(1)$

$(X_n=O_P(1))\wedge (Y_n\stackrel{P}{\to }0)\Rightarrow X_n{Y}_n\stackrel{P}{\to }0$

$(Y_n=O_P(1))\wedge (X_n=o_P(Y_n))\Rightarrow X_n\stackrel{P}{\to }0$

Link to original

Delta Method

Definition

Univariate Delta Method

Let $\{X_n\}$ be a sequence of random variables satisfying $\sqrt{n}(X_n-\theta )\stackrel{D}{\to }N(0,{\sigma }^2)$, $g$ be a differentiable function at $x=\theta$, and $g^{\prime }(\theta )\ne 0$, then $\sqrt{n}(g(X_n)-g(\theta ))\stackrel{D}{\to }{g}^{\prime }(\theta )N(0,{\sigma }^2)=N(0,{\sigma }^2{g}^{\prime }(\theta {)}^2)$

Proof

By Taylor Series approximation $g(X_n)=g(\theta )+g^{\prime }(\theta )(X_n-\theta )+o_P(|X_n-\theta |)$ $\Rightarrow \sqrt{n}(g(X_n)-g(\theta ))=g^{\prime }(\theta )\sqrt{n}(X_n-\theta )+o_P(\sqrt{n}|X_n-\theta |)$

Where $\sqrt{n}(X_n-\theta )\stackrel{D}{\to }N(0,{\sigma }^2)$ by the assumption. By the continuous mapping theorem, $\sqrt{n}|X_n-\theta |$, which is a function of $X_n$, also converges to random variable, so it is Boundedness in Probability $\sqrt{n}|X_n-\theta |=O_P(1)$ by the property of converging random vector

Therefore, $\sqrt{n}(g(X_n)-g(\theta ))=g^{\prime }(\theta )N(0,{\sigma }^2)+o_P(O_P(1))=g^{\prime }(\theta )N(0,{\sigma }^2)=N(0,{\sigma }^2{g}^{\prime }(\theta {)}^2)$ by the property of sequence of random variables bounded in probability

Examples

Estimation of the sample variance of Bernoulli Distribution

Let $X_1,X_2,\dots ,X_n\sim Ber(p)$ $\sqrt{n}({\bar{X}}_n-p)\to N(0,p(1-p))$ by CLT

$\sqrt{n}(g({\bar{X}}_n)-g(p))\to g^{\prime }(p)N(0,p(1-p))$ by Delta method

Let $g(x)=x(1-x)$, then $\sqrt{n}({\bar{X}}_n(1-{\bar{X}}_n)-p(1-p))\to N(0,p(1-p)(1-2p{)}^2)$

Therefore, the sample mean and variance follow such distributions ${\bar{X}}_n\to N\left(p,\frac{p(1-p)}{n}\right)=N\left(\mu ,\frac{\sigma }{n}\right)$ ${\bar{X}}_n(1-{\bar{X}}_n)=\hat{\sigma }\to N\left(p(1-p),\frac{p(1-p)(1-2p{)}^2}{n}\right)=N\left(\sigma ,\frac{\sigma (1-2p{)}^2}{n}\right)$

Visualization

• x-axis: $\mu =p$
• y_axis: ${\sigma }^2=p(1-p)$
• y1: $y=p(1-p)$^[variance by mean]
• y2: ${\bar{X}}_n$^[sample mean]
• y3, y4: ${\bar{X}}_n(1-{\bar{X}}_n)$^[sample variance that calculated by the sample mean]
• y5: first order approximated line at $(p,p(1-p))$

If sample size $n\to \infty$, variance of sample mean $V({\bar{X}}_n)\to 0$. So, sample variance $\hat{\sigma }$ can be well approximated by first-order approximation.

Link to original

MGF Technique

Definition

Let $\{X_n\}$ be a Sequence of random variables with MGF $M_{X_n}(t)$ and $X$ be a Random Variable with MGF $M_X(t)$, then $\underset{n\to \infty }{\lim }{M}_{X_n}(t)=M_X(t)\iff X_n\stackrel{D}{\to }X$

A technique calculating limiting distribution using Moment Generating Function

Facts

$\underset{n\to \infty }{\lim }\psi (n)=0\Rightarrow \underset{n\to \infty }{\lim }(1+\frac{b}{n}+\frac{\psi (n)}{n}{)}^{cn}=e^{bc}$

Link to original

Central Limit Theorem

Central Limit Theorem

Definition

Let $X_1,\dots ,X_n$ be i.i.d. Random Sample from a distribution with mean $\mu$, variance ${\sigma }^2$, then $\frac{\overline{X}-\mu }{\sigma /\sqrt{n}}\stackrel{D}{\to }Z\sim N(0,1)$

For i.i.d random variables that have finite variance, the sample mean converges to Normal Distribution.

Link to original

Asymptotics for Multivariate Distributions

Asymptotics for Multivariate Distributions

Definition

Convergence in Probability

${\mathbf{X}}_n\stackrel{P}{\to }\mathbf{X}\iff (\forall ϵ>0,\underset{n\to \infty }{\lim }P(||{\mathbf{X}}_n-\mathbf{X}||\ge ϵ)=0)$

Let $\{{\mathbf{X}}_n\}$ be a Sequence of random vectors and $\mathbf{X}$ be a Random Vector, then ${\mathbf{X}}_n$ converges in probability to $\mathbf{X}$ if $\forall ϵ>0,\underset{n\to \infty }{\lim }P(||{\mathbf{X}}_n-\mathbf{X}||\ge ϵ)=0$, and denoted by ${\mathbf{X}}_n\stackrel{P}{\to }\mathbf{X}$

Let $\{{\mathbf{X}}_n\}$ be a Sequence of random vectors and $\mathbf{X}$ be a Random Vector, then $(\forall j\in \{1,2,\dots ,p\},X_{nj}\stackrel{P}{\to }{X}_j)\iff {\mathbf{X}}_n\stackrel{P}{\to }\mathbf{X}$

Convergence in Distribution

${\mathbf{X}}_n\stackrel{D}{\to }\mathbf{X}\iff (\forall \mathbf{x}\in C(F_{\mathbf{X}}),\underset{n\to \infty }{\lim }{F}_{{\mathbf{X}}_n}(\mathbf{x})=F_{\mathbf{X}}(\mathbf{x}))$

Let $\{{\mathbf{X}}_n\}$ be a Sequence of random vectors with CDF $F_{{\mathbf{X}}_n}$, $\mathbf{X}$ be a Random Variable with cdf $F_{\mathbf{X}}$, and $C(F_{\mathbf{X}})$ be a set of every point at which $F_X$ is continuous, then $\{{\mathbf{X}}_n\}$ converges in distribution to $\mathbf{X}$ if $\forall \mathbf{x}\in C(F_{\mathbf{X}}),\underset{n\to \infty }{\lim }{F}_{{\mathbf{X}}_n}(\mathbf{x})=F_{\mathbf{X}}(\mathbf{x})$, and denoted by ${\mathbf{X}}_n\stackrel{D}{\to }\mathbf{X}$. $\mathbf{X}$ is also called limiting distribution of ${\mathbf{X}}_n$ or asymptotic distribution of ${\mathbf{X}}_n$

Continuous Mapping Theorem

Let $g$ be a Continuous Function ${\mathbf{X}}_n\stackrel{D}{\to }\mathbf{X}\Rightarrow g({\mathbf{X}}_n)\stackrel{D}{\to }g(\mathbf{X})$

MGF Technique

Let $\{{\mathbf{X}}_n\}$ be a Sequence of random vectors with MGF $M_{{\mathbf{X}}_n}(t)$ and $\mathbf{X}$ be a Random Vector with MGF $M_{\mathbf{X}}(t)$, then $\underset{n\to \infty }{\lim }{M}_{{\mathbf{X}}_n}(t)=M_{\mathbf{X}}(t)\iff {\mathbf{X}}_n\stackrel{D}{\to }\mathbf{X}$

Central Limit Theorem

Let $\{{\mathbf{X}}_n\}$ be i.i.d. Sequence of random vectors from a distribution with mean $E({\mathbf{X}}_1)=\mathbf{\mu }$, variance-covariance matrix $\mathrm{Var}({\mathbf{X}}_1)=\Sigma$, then $\sqrt{n}(\overline{\mathbf{X}}-\mathbf{\mu })\stackrel{D}{\to }{N}_p(0,\Sigma )$

For i.i.d random variables that have finite variance, the sample mean converges to Normal Distribution.

Delta Method

Let $\{{\mathbf{X}}_n\}$ be a Sequence of p-dimensional random vectors with $\sqrt{n}({\mathbf{X}}_n-\mathbf{\mu })\stackrel{D}{\to }{N}_p(0,\Sigma )$, $\mathbf{g}:{\mathbb{R}}^p\to {\mathbb{R}}^k$, $B=\left(\frac{\partial g_i}{\partial {\mu }_i}\right)$ be a $k\times p$ matrix, and $B(\mathbf{\mu })\ne 0$, then $\sqrt{n}(\mathbf{g}({\mathbf{X}}_n)-\mathbf{g}(\mathbf{\mu }))\stackrel{D}{\to }{N}_k(0,B\Sigma B^{⊺})$

Facts

Let ${\mathbf{v}}^{⊺}=(v_1,v_2,\dots ,v_p)\in {\mathbb{R}}^p$, then $\forall j\in \{1,2,\dots p\},|v_j|\le ||\mathbf{v}||\le \sum _{i=1}^n|v_i|$

Let $\{{\mathbf{X}}_n\}$ be a Sequence of p-dimensional random vectors, ${\mathbf{X}}_n\stackrel{D}{\to }N(\mathbf{\mu },\Sigma )$, $A$ be a $m\times p$ constant matrix, and $\mathbf{b}$ be a $m$ dimensional constant vector, then $A{\mathbf{X}}_n+\mathbf{b}\stackrel{D}{\to }N(A\mathbf{\mu }+\mathbf{b},A\Sigma A^{⊺})$

Link to original

Some Elementary Statistical Inferences

Sample and Statistic

Random Sample

Definition

A realization of i.i.d. Random Vector

Properties

Let $\mathbf{x}=(X_1,X_2,\dots ,X_m{)}^{⊺},\mathbf{y}=(Y_1,Y_2,\dots ,Y_n{)}^{⊺}$ be random samples, $A$ be $k\times m$ constant matrix, and $B$ be $p\times n$ constant matrix, then the Expected Value and Covariance Matrix of random vectors become

• $E(A\mathbf{x})=AE(\mathbf{x})$
• $\mathrm{Var}(A\mathbf{x})=A\mathrm{Cov}(\mathbf{x})A^{⊺}$
• $\mathrm{Cov}(A\mathbf{x},B\mathbf{y})=A\mathrm{Cov}(\mathbf{x},\mathbf{y})B^{⊺}$

Link to original

Statistic

Definition

A quantity computed from values in a Random Sample

Link to original

Bias of an Estimator

Definition

Let $X_1,X_2,\dots ,X_n$ be a Random Sample from $f(\mathbf{x}|\mathbf{\theta })$ where $\mathbf{\theta }\in \Omega$ is a parameter, and $\hat{\mathbf{\theta }}=T(X_1,\dots ,X_n)$ be a Statistic. $\mathrm{Bias}(\hat{\mathbf{\theta }})=E((\hat{\mathbf{\theta }}))-\mathbf{\theta }$

$\forall \mathbf{\theta }\in \Omega ,E(\hat{\mathbf{\theta }})=\mathbf{\theta }\Rightarrow \hat{\mathbf{\theta }}$ is unbiased estimator An estimator is unbiased if its bias is equal to zero for all values of parameter $\mathbf{\theta }$.

Link to original

Order Statistic

Order Statistic

Definition

Let $X_1,X_2,\dots ,X_n$ be Random Sample from a population with a PDF $f(x)$ and CDF $F(x)$. And let $Y_1$ be the smallest of $X_i$‘s, $Y_2$ be the 2nd smallest of $X_i$‘s , $\dots$, and $Y_n$ be the largest of $X_i$‘s, then $Y_1<Y_2<\cdots <Y_n$ is called the order statistic of $X_1,X_2,\dots ,X_n$

Properties

Joint PDF

$g(y_1,y_2,\dots ,y_n)=n!{\prod }_{i=1}^{n}f(y_i)$ where $y_1<y_2<\cdots y_n$

Marginal PDF

$g_k(y_k)=\frac{n!}{(k-1)!(n-k)!}(F(y_k){)}^{k-1}(1-F(y_k){)}^{n-k}f(y_k)$

Joint PDF of $Y_i$ and $Y_j$

Suppose $i<j$ $g_{ij}(y_i,y_j)=\frac{n!}{(i-1)!(j-i-1)!(n-j)!}(F(y_i){)}^{i-1}(F(y_j)-F(y_i){)}^{j-i-1}(1-F(y_j){)}^{n-j}f(y_i)f(y_j)$

Joint PDF of $Y_{l_1},Y_{l_2},\dots ,Y_{l_k}$

Suppose $l_1<l_2<\cdots <l_k$ $g_{l_1,l_2,\dots ,l_k}(y_{l_1},y_{l_2},\dots ,y_{l_k})=\frac{n!}{{\prod }_{i=1}^{k+1}(l_i-l_{i-1}-1)!}\prod _{i=1}^{k}f(y_{l_i})\prod _{i=1}^{k+1}[F(y_{l_i})-F(y_{l_{i-1}}){]}^{l_i-l_{i-1}-1}$ where $l_0=0,l_{k+1}=n+1,x_{(l_0)}=-\infty ,x_{(l_{k+1})}=\infty$

Link to original

Quantiles

Definition

Let $X$ be a Random Variable with cdf $F(x)$, and ${\xi }_p=F^{-1}(p)$ be a $p$-th quantile of $X$

Properties

Estimator of Quantile

Let $Y_1<\cdots <Y_n$ be order statistics, then we can define an estimator of quantile using the order statistic ${\xi }_p=Y_k$ where $k=\lfloor (n+1)p\rfloor$

and call the $Y_k$ a $p$-th sample quantile

Confidence Intervals of Quantiles

$P(Y_i<{\xi }_p<Y_j)=\sum _{k=i}^{j-1}\left(\genfrac{}{}{0ex}{}{n}{k}\right)p^k(1-p{)}^{n-k}=\gamma$ where $i<\lfloor (n+1)p\rfloor <j$

Select $i,j$ satisfying the equation

Link to original

Tolerance Limits for Distributions

Tolerance Limits for Distributions

Definition

Let $X_1,\dots ,X_n$ be Random Sample from a distribution with cdf $F(x)$, and $Y_1<\cdots <Y_n$ be Order Statistic, then If $P[F(Y_i)-F(Y_j)\ge p]=\gamma$, then $(Y_i,Y_j)$ is $100\gamma \%$ tolerance limits for $100p\%$ of the probability for the distribution of $X$

Properties

Joint PDF

$h(z_1,z_2,\dots ,z_n)=\{\begin{array}{ll}n!& 0<z_1<z_2<\cdots <z_n<1\\ 0& \text{otherwise}\end{array}$

Computation of $\gamma$

$P(Z_i-Z_j\ge p)=P(Z_{j-i}\ge p)=\gamma ={\int }_p^1{h}_{j-i}(v)dv$ where $h_k(v)=\frac{n!}{(k-1)!(n-k)!}{v}^{k-1}(1-v{)}^{n-k}$

A probability that the tolerance interval $(Z_j,Z_i)$ contains $100p\%$ of the probability

Link to original

Introduction to Hypothesis Testing

Hypothesis Testing

Definition

Types of Errors

	$H_0$ is true	$H_1$ is true
Not reeject $H_0$	$1-\alpha$	type 2 error ($\beta$)
Reject $H_0$	type 1 error ($\alpha$)	$1-\beta$

Let the set $C$ where $H_0$ is rejected be a rejection region and a statistic $T$ for testing be a test statistic, then the size $\alpha$ is the maximum of probability of type 1 error a test of size $\alpha$ is $\max P_{H_0}(T\in C)\le \alpha$ a power of the test is $P_{H_1}(T\in C)=1-\beta$ the probability of the type 2 error is $P_{H_1}(T\notin C)=\beta$

p-value

The probability of obtaining test results at least as extreme as the result of actually observed under $H_0$

Link to original

Goodness-of-Fit Test for Multinomial Distribution

Definition

Let $\mathbf{X}\sim \mathrm{Mult}(n,p_1,\dots ,p_{c-1})$ be a Multinomial Distribution and consider a null hypothesis $H_0:p_1=P_{10},p_2=P_{20},\dots ,p_c=P_{c0}$. Then, the test Statistic, which follows Chi-squared Distribution, is defined as $Q_{c-1}=\sum _{i=1}^c\frac{(X_i-n{p}_i{)}^2}{n{p}_i}\stackrel{D}{\to }{\chi }^2(c-1)$

Link to original

Homogeneity Test for Multinomial Distribution

Definition

Let ${\mathbf{X}}_1\sim \mathrm{Mult}(n_1,p_{11},\dots ,p_{c1}),{\mathbf{X}}_2\sim \mathrm{Mult}(n_2,p_{12},\dots ,p_{c2}),\dots ,{\mathbf{X}}_r\sim \mathrm{Mult}(n_r,p_{1r},\dots ,p_{cr})$ and consider a null hypothesis $H_0:p_{11}=p_{12}=\cdots =p_{1r},p_{21}=p_{22}=\cdots =p_{2r},p_{c1}=p_{c2}=\cdots =p_{cr}$. Then, the test Statistic, which follows Chi-squared Distribution, is defined as $Q=\sum _{j=1}^r\sum _{i=1}^c\frac{(X_{ij}-n_j{\hat{p}}_{ij}{)}^2}{n_j{\hat{p}}_{ij}}\stackrel{D}{\to }{\chi }^2((r-1)(c-1))$ where ${\hat{p}}_{ij}=\frac{{\sum }_{j=1}^r{X}_{ij}}{{\sum }_{j=1}^r{n}_j}$

Link to original

Independence Test for Two Discrete Variables

Definition

Let category variables $A_1,\dots ,A_r$ and $B_1,\dots ,B_c$ and consider a null hypothesis $H_0:A,B$ are independent. Then, the test Statistic, which follows Chi-squared Distribution, is defined as $Q=\sum _{j=1}^b\sum _{i=1}^a\frac{(X_{ij}-n{\hat{p}}_{ij}{)}^2}{n{\hat{p}}_{ij}}\stackrel{D}{\to }{\chi }^2((r-1)(c-1))$ where $n=\sum _j\sum _i{X}_{ij}$, ${\hat{p}}_{ij}={\hat{p}}_{i.}{\hat{p}}_{.j}=\frac{X_{i.}}{n}\frac{X_{.j}}{n}$, $X_i=\sum _j{X}_{ij}$, $X_j=\sum _i{X}_{ij}$

Link to original

The Method of Monte Carlo

Inverse Transform Sampling

Definition

$X=F_X^{-1}(U)$

we can generate $X$ with CDF $F_X(x)$ by using inverse function of CDF $F_X^{-1}$ and uniform random variable $U\sim u[0,1]$

Facts

Let $U\sim u(0,1)$ and $F$ be continuous CDF, then $F^{-1}(U)\sim F$

Link to original

Box-Muller Transformation

Definition

Let $U_1,U_2\stackrel{iid}{\sim }U(0,1)$, $R^2/2\sim Exp(1)$, and $\theta \sim U(0,2\pi )$, then $X_1=R\cos (\theta )=\sqrt{-2log{U}_1}\cos (2\pi U_2)\sim \mathrm{N}(0,1)$ and $X_2=R\sin (\theta )=\sqrt{-2log{U}_1}\sin (2\pi U_2)\sim \mathrm{N}(0,1)$

The method that generates pairs of i.i.d. normal distribution by using two uniform distributions or Exponential Distribution

Link to original

Accept Reject Generation Algorithm

Definition

Let $Y\sim g(y),U\sim u(0,1)$ be independent random variables, $f(x)$ be a PDF which we want to sample, and $M0,\frac{f(x)}{g(x)}\le M$ step 1: Generate $Y$ and $U\sim u(0,1)$. step 2: If $U\le \frac{f(y)}{Mg(y)}\Rightarrow X:=Y$ otherwise, go to step 1. step 3. repeat step 1 and 2.

Link to original

Bootstrap Procedures

Bootstrapping

Definition

Let $X\sim f(x|\theta )$ be a random variable with $\theta \in \Omega$, $\mathbf{X}=(X_1,X_2,\dots ,X_n)$ be a Random Sample of $X$, and $\hat{\theta }=\hat{\theta }(\mathbf{X})$ be a point estimator of $\theta$ Then, the bootstrap sample is $n$-dimensional sample vector ${\mathbf{X}}^{\ast }=(X_1^{\ast },X_2^{\ast },\dots ,X_n^{\ast })$ drawn with replacement from the vector of original samples $\mathbf{X}=(X_1,X_2,\dots ,X_n)$

Bootstrap Confidence Interval

Draw $B$ (large number) bootstrap samples ${\mathbf{X}}_1^{\ast },{\mathbf{X}}_2^{\ast },\dots ,{\mathbf{X}}_B^{\ast }$ and calculate confidence interval using the samples Calculate point estimators using the bootstrap samples ${\hat{\theta }}_j^{\ast }:=\hat{\theta }({\mathbf{X}}_j^{\ast })$, and define order statistics ${\hat{\theta }}_{(1)}^{\ast }\le {\hat{\theta }}_{(2)}^{\ast }\le \cdots \le {\hat{\theta }}_{(B)}^{\ast }$ for the estimators. Then, $100(1-\alpha )\%$ confidence bootstrap confidence interval for $\theta$ is $({\hat{\theta }}_{(m)}^{\ast },{\hat{\theta }}_{(B+1-m)}^{\ast })$, where $m=\lfloor \frac{\alpha }{2}b\rfloor$

Link to original

Maximum Likelihood Methods

Maximum Likelihood Estimation

Likelihood Function

Definition

$L(\theta |\mathbf{x}):=L(\theta )={\prod }_{i=1}^{n}f(x_i|\theta )$

Log-Likelihood Function

$l(\theta ):=\ln L(\theta )={\sum }_{i=1}^n\ln f(x_i|\theta )$

Link to original

Maximum Likelihood Estimation

Definition

MLE is the method of estimating the parameters of an assumed Distribution

Let $X_1,X_2,\dots ,X_n$ be Random Sample with PDF $f(x|\theta )$, where $\theta \in \Omega$, then the MLE ${\hat{\theta }}_{\text{MLE}}$ of $\theta$ is estimated as ${\hat{\theta }}_{\text{MLE}}=\underset{\theta }{\mathrm{argmax}}L(\theta |\mathbf{x})$

Regularity Conditions

• R0: The pdfs are distinct, i.e. $\theta \ne {\theta }^{\prime }\Rightarrow f(x_i|\theta )\ne f(x_i|{\theta }^{\prime })$
• R1: The pdfs have same supports $\forall \theta$
• R2: The true value ${\theta }_0$ is an interior point in $\Omega$
• R3: The pdf $f(x|\theta )$ is twice differentiable with respect to $\theta \in \Omega$
• R4: $\frac{\partial }{\partial {\theta }^2}\int f(x|\theta )dx=\int \frac{\partial }{\partial {\theta }^2}f(x|\theta )dx$
• R5: The pdf $f(x|\theta )$ is three times differentiable with respect to $\theta \in \Omega$, $\forall \theta \in \Omega ,\left|\frac{{\partial }^3}{\partial {\theta }^3}\ln f(x|\theta )\right|\le M(x)$, and $\exists c\in \mathbb{R},\exists M(x),\forall |\theta -{\theta }_0|<c,\forall$ interior point $x,E_{{\theta }_0}[M(X)]<\infty$

Properties

Functional Invariance

If $\hat{\theta }$ is the MLE for $\theta$, then $g(\hat{\theta })$ is the MLE of $g(\theta )$

Consistency

Under R0 ~ R2 Regularity Conditions, let ${\theta }_0$ be a true parameter, $f(x|\theta )$ is differentiable with respect to $\theta \in \Omega$, then $\frac{\partial }{\partial \theta }L(\theta )=0$ has a solution ${\hat{\theta }}_n$ such that ${\hat{\theta }}_n\stackrel{P}{\to }{\theta }_0$

Asymptotic Normality

Under the R0 ~ R5 Regularity Conditions, let $X_1,X_2,\dots ,X_n$ be Random Sample with PDF $f(x|\theta )$, where $\theta \in \Omega$, ${\hat{\theta }}_n$ be a consistent Sequence of solutions of MLE equation $\frac{\partial l(\theta )}{\partial \theta }=0$, and $0<I({\theta }_0)<\infty$, then $\sqrt{n}({\hat{\theta }}_n-{\theta }_0)\stackrel{D}{\to }N\left(0,\frac{1}{I({\theta }_0)}\right)$ where $I({\theta }_0)$ is the Fisher Information.

By the asymptotic normality, the MLE estimator is asymptotically efficient under R0 ~ R5 Regularity Conditions

Asymptotic Confidence Interval

By the asymptotic normality of MLE, $\sqrt{nI(\hat{\theta })}(\hat{\theta }-\theta )\stackrel{D}{\to }N(0,1)$ Thus, $100(1-\alpha )\%$ confidence interval of for $\theta$ is $\left(\hat{\theta }-z_{\alpha /2}\frac{1}{\sqrt{nI(\hat{\theta })}},\hat{\theta }+z_{\alpha /2}\frac{1}{\sqrt{nI(\hat{\theta })}}\right)$

Delta method for MLE Estimator

Under the R0 ~ R5 Regularity Conditions, let $g(x)$ be a continuous function and $g^{\prime }({\theta }_0)\ne 0$, then $\sqrt{n}(g({\hat{\theta }}_n)-g({\theta }_0))\stackrel{D}{\to }N\left(0,\frac{g^{\prime }({\theta }_0{)}^2}{I({\theta }_0)}\right)$

Facts

Under R0 and R1 regularity conditions, let ${\theta }_0$ be a true parameter, then $\forall \theta \ne {\theta }_0,\underset{n\to \infty }{\lim }{P}_{{\theta }_0}[L({\theta }_0)>L(\theta )]=1$

Link to original

Rao-Cramer Lower Bound and Efficiency

Bartlett Identities

Definition

First Bartlett Identity

$0=E\left[\frac{\partial \ln f(X|\theta )}{\partial \theta }\right]=E[s(\theta |x)]$ where $f(X|\theta )$ is a Likelihood Function and $s(\theta |x)$ is a Score Function

Second Bartlett Identity

$0=E\left[\frac{{\partial }^2\ln f(X|\theta )}{\partial {\theta }^2}\right]+E\left[{\left(\frac{\partial \ln f(X|\theta )}{\partial \theta }\right)}^2\right]$ where $f(X|\theta )$ is a Likelihood Function

Link to original

Score Function

Definition

$s(\theta |x):=\frac{\partial \ln L(\theta |x)}{\partial \theta }$

The gradient of the log-likelihood function with respect to the parameter vector. The score indicates the steepness of the log-likelihood function

Facts

The score will vanish at a local Extremum

Link to original

Fisher Information

Definition

Fisher Information

I(\theta) &:= E\left[ \left( \frac{\partial \ln f(X|\theta)}{\partial \theta} \right)^{2} \right] = \int_{\mathbb{R}} \left( \frac{\partial \ln f(X|\theta)}{\partial \theta} \right)^{2} p(x, \theta)dx\\ &= -E\left[ \frac{\partial^{2} \ln f(X|\theta)}{\partial \theta^{2}} \right] = -\int_{\mathbb{R}} \frac{\partial^{2} \ln f(X|\theta)}{\partial \theta^{2}} p(x, \theta)dx\\ \end{aligned}$$ by the [[Bartlett Identities#second-bartlett-identity|second Bartlett identity]] $$I(\theta) = \operatorname{Var}\left( \frac{\partial \ln f(X|\theta)}{\partial \theta} \right) = \operatorname{Var}(s(\theta|x))$$ where $s(\theta|x)$ is a [[Score Function]] ## Fisher Information Matrix ![[Pasted image 20231224171415.png|800]] Let $\mathbf{X}$ be a [[Random Vector]] with [[Density Function|PDF]] $f(x|\boldsymbol{\theta})$, where $\boldsymbol{\theta} \in \Omega \subset R^{p}$, then the **Fisher information matrix** for on $\boldsymbol{\theta}$ is a $p \times p$ matrix defined as $$I(\boldsymbol{\theta}) := \operatorname{Cov}\left( \cfrac{\partial}{\partial \boldsymbol{\theta}} \ln f(x|\boldsymbol{\theta}) \right) = E\left[ \left( \cfrac{\partial}{\partial \boldsymbol{\theta}} \ln f(x|\boldsymbol{\theta}) \right) \left( \cfrac{\partial}{\partial \boldsymbol{\theta}} \ln f(x|\boldsymbol{\theta}) \right)^\intercal \right] = -E\left[ \cfrac{\partial^{2}}{\partial \boldsymbol{\theta} \partial \boldsymbol{\theta}^\intercal} \ln f(x|\boldsymbol{\theta}) \right]$$ and $jk$-th element of $I(\boldsymbol{\theta})$, $I_{jk} = - E\left[ \cfrac{\partial^{2}}{\partial \theta_{j} \partial\theta_{k}} \ln f(x|\boldsymbol{\theta}) \right]$ # Properties ## Chain Rule The information in length $n$ [[Random Sample]] $X_{1}, X_{2}, \dots, X_{n}$ is $n$ times the information in a single sample $X_{i}$ $I_\mathbf{X}(\theta) = n I_{X_{1}}(\theta)$ # Facts > In a location model, information is not dependent on a location parameter. > $$I(\theta) = \int_{-\infty}^{\infty}\left( \frac{f'(z)}{f(z)} \right)^{2} f(z)dz$$Link to original

Rao-Cramer Lower Bound

Definition

Let $X_1,X_2,\dots ,X_n$ be Random Sample with PDF $f(x|\theta )$ with the R0 ~ R4 regularity conditions and $Y=u(X_1,X_2,\dots ,X_n)$ be a Statistic with $E(Y)=k(\theta )$, then $\mathrm{Var}(Y)\ge \frac{k^{\prime }(\theta {)}^2}{nI(\theta )}$

Facts

Let $Y=u(X_1,X_2,\dots ,X_n)$ be an unbiased estimator of $\theta$, then $E(Y)=k(\theta )=\theta$. Thus, by a Rao-Cramer Lower Bound, $\mathrm{Var}(Y)\ge \frac{1}{nI(\theta )}$

Link to original

Efficiency

Definition

Efficient Estimator

$\mathrm{Var}(Y)=\frac{1}{nI(\theta )}$ where $I(\theta )$ is a Fisher Information

An unbiased estimator $Y$ is called an efficient estimator if its variance is the Rao-Cramer Lower Bound

Efficiency

Let $Y$ be an unbiased estimator, then the efficiency of $Y$, $0\le e(Y)\le 1$ is $e(Y)=\frac{1/nI(\theta )}{\mathrm{Var}(Y)}$

Asymptotic Efficiency

Let $X_1,X_2,\dots ,X_n$ be i.i.d. random variables with PDF $f(x|\theta )$, and ${\hat{\theta }}_1$ be an estimator satisfying $\sqrt{n}({\hat{\theta }}_1-{\theta }_0)\stackrel{D}{\to }N(0,{\sigma }_1^2)$, then

• the asymptotic efficiency of ${\hat{\theta }}_1$ is $e({\hat{\theta }}_1)=\frac{1/I({\theta }_0)}{{\sigma }_1^2}$
• If $e({\hat{\theta }}_1)=1$, then ${\hat{\theta }}_1$ is called asymptotically efficient
• Assume that another estimator${\hat{\theta }}_2$ such that $\sqrt{n}({\hat{\theta }}_2-{\theta }_0)\stackrel{D}{\to }N(0,{\sigma }_2^2)$ Then, the asymptotic relative efficiency (ARE) of ${\hat{\theta }}_1$ to ${\hat{\theta }}_2$ is $e({\hat{\theta }}_1,{\hat{\theta }}_2)=\frac{{\sigma }_2^2}{{\sigma }_1^2}$

Examples

In a Laplace distribution, ${\sigma }_m^2=1,{\sigma }_{\overline{X}}^2=2$ So, sample median is $2$ times more efficient than sample mean for the estimator of the location parameter.

In a Normal distribution, ${\sigma }_m^2=\frac{\pi }{2},{\sigma }_{\overline{X}}^2=1$ So, sample mean is $\frac{\pi }{2}$ times more efficient than sample median for the estimator of the location parameter.

Link to original

Newton's Method

Definition

An iterative algorithm for finding the roots of a differentiable function, which are solution to the equation $f(x)=0$

Algorithm

Find the next point such that the Taylor series of the given point is 0 Taylor first approximation: $f(x)\simeq f(x_n)+f^{\prime }(x_n)(x-x_n)$ The point such that the Taylor series is 0: $x_{n+1}=x_n-\frac{f(x_n)}{f^{\prime }(x_n)}$

multivariate version: ${\mathbf{x}}_{n+1}=f({\mathbf{x}}_n)-\nabla f({\mathbf{x}}_n{)}^{-1}f({\mathbf{x}}_n)$

In convex optimization,

Find the minimum point^[its derivative is 0] of Taylor quadratic approximation. Taylor quadratic approximation: $f(x)\simeq f(x_n)+f^{\prime }(x_n)(x-x_n)+\frac{f^{\prime \prime }(x_n)}{2}(x-x_n{)}^2$ The derivative of the quadratic approximation: $f^{\prime }(x_n)+f^{\prime \prime }(x_n)(x-x_n)$ The minimum point of the quadratic approximation^[the point such that the derivative of the quadratic approximation is 0]: $x_{n+1}=x_n-\frac{f^{\prime }(x_n)}{f^{\prime \prime }(x_n)}$ multivariate version: ${\mathbf{x}}_{n+1}={\mathbf{x}}_n-{\nabla }^{2}f({\mathbf{x}}_n{)}^{-1}\nabla f({\mathbf{x}}_n)$

Examples

Solution of a linear system

Solve $A\mathbf{x}=\mathbf{b}$ with an MSE loss

The cost function is $f(\mathbf{x})=(A\mathbf{x}-\mathbf{b}{)}^{⊺}(A\mathbf{x}-\mathbf{b})$ and its gradient and hessian are $\nabla f(x)=-2(\mathbf{b}-A\mathbf{x}{)}^{⊺}$, ${\nabla }^{2}f(x)=2A^{⊺}A$

Then, solution is ${\mathbf{x}}_{n+1}={\mathbf{x}}_n+(A^{⊺}A{)}^{-1}{A}^{⊺}(\mathbf{b}-A{\mathbf{x}}_n)$ If $(A^{⊺}A)$ is invertible, ${\mathbf{x}}_{n+1}=(A^{⊺}A{)}^{-1}{A}^{⊺}\mathbf{b}$ is a Least Square solution.

Link to original

Fisher's Scoring Method

Definition

Fisher’s scoring method is a variation of Newton–Raphson method that uses Fisher Information instead of the Hessian Matrix

${\hat{\theta }}_{n+1}={\hat{\theta }}_n-\frac{l^{\prime }({\hat{\theta }}_n)}{I({\hat{\theta }}_n)}$ where $I(\theta )=E[l^{\prime \prime }(\theta )]$ is the Fisher Information.

Link to original

Maximum Likelihood Tests

Likelihood Ratio Test

Definition

Let $X_1,X_2,\dots ,X_n$ be Random Sample with PDF $f(x|\mathbf{\theta })$ where $\mathbf{\theta }\in \Omega$, $H_0:\mathbf{\theta }\in \omega ,H_1:\mathbf{\theta }\in \Omega \cap {\omega }^{\complement }$, where $\omega \subset \Omega$. $\Lambda =\frac{L(\hat{\omega })}{L(\hat{\Omega })}$ is called a likelihood ratio, where $\hat{\omega },\hat{\Omega }$ are MLE under $\omega$ and $\Omega$ respectively. If $H_0$ is true, then $\Lambda$ will be close to $1$, while if $H_0$ is not true, then the $\Lambda$ will be close to $0$

Therefore, LRT rejects $H_0$ when $\Lambda \le c$, where the $c$ is determined by the level alpha condition. $\alpha =P_{\omega }(\Lambda \le c)$

By the Wilks’ Theorem, the likelihood ratio test Statistic for the hypothesis $H_0:\theta ={\theta }_0,H_1:\theta \ne {\theta }_0$ is given by ${\chi }_L^2={\lambda }_{LR}:=-2\ln \Lambda =-2\ln \frac{L({\theta }_0)}{L(\hat{\theta })}\stackrel{D}{\to }{\chi }^2(1)$

Visualization

The comparison of the Likelihood Ratio Test, Wald Test, and Score Test

Link to original

Wilks' Theorem

Definition

Single Parameter

Under R0 ~ R5 regularity conditions, let $H_0:\theta ={\theta }_0$, $-2\ln \Lambda \stackrel{D}{\to }{\chi }^2(1)$ where $\Lambda =\frac{L({\theta }_0)}{L(\hat{\theta })}$ is the likelihood ratio, and the $\hat{\theta }$ is the MLE of $\theta$

Multiparameter

Under R0 ~ R5 regularity conditions, let $H_0:\mathbf{\theta }\in \omega$, $-2\ln \Lambda \stackrel{D}{\to }{\chi }^2(q)$ where $\Lambda =\frac{L(\hat{\omega })}{L(\hat{\Omega })}$ is the likelihood ratio, and the $\hat{\Omega }$ is the MLE of $\mathbf{\theta }$ without constraints.

Link to original

Wald Test

Definition

the Wald test Statistic for the hypothesis $H_0:\theta ={\theta }_0,H_1:\theta \ne {\theta }_0$ is given by ${\chi }_W^2={\left(\sqrt{nI(\hat{\theta })}(\hat{\theta }-{\theta }_0)\right)}^2\stackrel{D}{\to }{\chi }^2(1)$

Link to original

Score Test

Definition

the Score test Statistic for the hypothesis $H_0:\theta ={\theta }_0,H_1:\theta \ne {\theta }_0$ is given by ${\chi }_S^2={\left(\frac{l^{\prime }({\theta }_0)}{\sqrt{nI({\theta }_0)}}\right)}^2\stackrel{D}{\to }{\chi }^2(1)$

Link to original

Multiparameter Case

Multiparametric Maximum Likelihood Estimation

Definition

Multiparameter MLE

Let $X_1,X_2,\dots ,X_n$ be Random Sample with PDF $f(x|\mathbf{\theta })$, where $\mathbf{\theta }=({\theta }_1,{\theta }_2,\dots ,{\theta }_p{)}^{⊺}\in \Omega \subset R^p$, then the MLE ${\hat{\mathbf{\theta }}}_{\text{MLE}}$ of $\mathbf{\theta }$ is estimated as ${\hat{\mathbf{\theta }}}_{\text{MLE}}=\underset{\mathbf{\theta }}{\mathrm{argmax}}L(\mathbf{\theta }|\mathbf{x})$

Properties

Multiparameter Asymptotic Normality

Under the R0 ~ R5 Regularity Conditions, let $X_1,X_2,\dots ,X_n$ be Random Sample with PDF $f(x|\mathbf{\theta })$, where $\mathbf{\theta }=({\theta }_1,{\theta }_2,\dots ,{\theta }_p{)}^{⊺}\in \Omega \subset R^p$, then

• $\frac{\partial l(\mathbf{\theta })}{\partial \mathbf{\theta }}=0$ has solution ${\hat{\mathbf{\theta }}}_n$ such that ${\hat{\mathbf{\theta }}}_n\stackrel{P}{\to }\mathbf{\theta }$
• $\sqrt{n}({\hat{\mathbf{\theta }}}_n-\mathbf{\theta })\stackrel{D}{\to }{N}_p(0,I^{-1}(\mathbf{\theta }))$
• $\forall j\in \{1,2,\dots ,p\},\sqrt{n}({\hat{\theta }}_{nj}-{\theta }_j)\stackrel{D}{\to }N(0,I^{-1}(\mathbf{\theta }{)}_{jj})$

Delta Method for Multiparameter MLE

Multiparameter Rao-Cramer Lower Bound

Let $X_1,X_2,\dots ,X_n$ be Random Sample with PDF $f(x|\mathbf{\theta })$, $Y_j=u(X_1,X_2,\dots ,X_n)$ be an unbiased estimator of ${\theta }_j$, then $E(Y_j)=k(\theta )={\theta }_j$. Thus, by a Rao-Cramer Lower Bound, $\mathrm{Var}(Y_j)\ge \frac{1}{n}{I}^{-1}(\mathbf{\theta }{)}_{jj}$ where $I^{-1}(\mathbf{\theta }{)}_{jj}$ is $j$-th diagonal element of $I^{-1}(\mathbf{\theta })$

If $=$ holds, then $Y_j$ is called an efficient estimator of ${\theta }_j$

In a location-scale model, the information of the parameters do not depend on the locale parameter.

Link to original

Multiparametric Likelihood Ratio Test

Definition

Let $X_1,X_2,\dots ,X_n$ be Random Sample from PDF $f(x|\mathbf{\theta })$, where $\mathbf{\theta }\in \Omega \subset {\mathbb{R}}^p$, $p$ be the number of parameters, $q$ be the number of constraints (restrictions) $g_1(\mathbf{\theta })=a_1,\dots ,g_q(\mathbf{\theta })=a_q$, where $g_i$ is differentiable function $H_0:\mathbf{\theta }\in \omega ,H_1:\mathbf{\theta }\in \Omega \setminus \omega$, where the dimension of $\omega$ is $p-q$.

Now, consider a test statistic to test the hypothesis $\Lambda :=\frac{\underset{\theta \in \omega }{\max }L(\theta )}{\underset{\theta \in \Omega }{\max }L(\theta )}=\frac{L(\hat{\omega })}{L(\hat{\Omega })}$ If $H_0$ is true, then $\Lambda$ will be close to $1$, while if $H_0$ is not true, then the $\Lambda$ will be close to $0$ Therefore, LRT rejects $H_0$ when $\Lambda \le c$, where the $c$ is determined by the level alpha condition. $\alpha =P_{\omega }(\Lambda \le c)$

By the Wilks’ Theorem, the likelihood ratio test Statistic for the hypothesis $H_0:\mathbf{\theta }\in \omega ,H_1:\mathbf{\theta }\in \Omega \setminus \omega$ is given by ${\chi }_L^2:=-2\ln \Lambda =-2\ln \frac{L(\hat{\omega })}{L(\hat{\Omega })}\stackrel{D}{\to }{\chi }^2(q)$

Link to original

EM Algorithm

Expectation-Maximization Algorithm

Definition

Let $\mathbf{X}=(X_1,X_2,\dots ,X_n)$ be an observed data, $\mathbf{Z}=(Z_1,Z_2,\dots ,Z_n)$ be an unobserved (latent) variable, $\mathbf{X},\mathbf{Z}$ are independent, $g(\mathbf{X}|\mathbf{\theta })$ be a joint pdf of $\mathbf{X}$, $h(\mathbf{X},\mathbf{Z}|\mathbf{\theta })$ be a joint pdf of $\mathbf{X},\mathbf{Z}$, $k(\mathbf{Z}|\mathbf{X},\mathbf{\theta })$ be a conditional pdf of $\mathbf{Z}$ given $\mathbf{X}$

By the definition of a conditional pdf, we have the identity $k(\mathbf{Z}|\mathbf{X},\mathbf{\theta })=\frac{h(\mathbf{X},\mathbf{Z}|\mathbf{\theta })}{g(\mathbf{X}|\mathbf{\theta })}$

The goal of the EM algorithm is maximizing the observed likelihood $L(\mathbf{\theta }|\mathbf{X})=g(\mathbf{X}|\mathbf{\theta })$ using the complete likelihood $L^c(\mathbf{\theta }|\mathbf{X},\mathbf{Z})=h(\mathbf{X},\mathbf{Z}|\mathbf{\theta })$.

Using the definition conditional pdf, we derive the identity for an arbitrary but fixed ${\mathbf{\theta }}_0\in \Omega$

&= \int \ln[h(\mathbf{X}, \mathbf{Z} | \boldsymbol{\theta})] k(\mathbf{Z} | \mathbf{X}, \boldsymbol{\theta}_{0}) d \mathbf{Z} - \int \ln[k(\mathbf{Z} | \mathbf{X}, \boldsymbol{\theta})]k(\mathbf{Z} | \mathbf{X}, \boldsymbol{\theta}_{0})d\mathbf{Z}\\ &= E_{\boldsymbol{\theta}_{0}}[\ln L^{c}(\boldsymbol{\theta}|\mathbf{X}, \mathbf{Z}) | \boldsymbol{\theta}_{0}, \mathbf{X}] - E_{\boldsymbol{\theta}_{0}}[\ln k(\mathbf{Z} | \mathbf{X}, \boldsymbol{\theta}) | \boldsymbol{\theta}_{0}, \mathbf{X}] \end{aligned}$$ Let the first term of RHS be a quasi-likelihood function $$Q(\boldsymbol{\theta} | \boldsymbol{\theta}_{0}, \mathbf{X}) := E_{\boldsymbol{\theta}_{0}}[\ln L^{c}(\boldsymbol{\theta}|\mathbf{X}, \mathbf{Z}) | \boldsymbol{\theta}_{0}, \mathbf{X}]$$ EM algorithm maximizes $Q(\boldsymbol{\theta} | \boldsymbol{\theta}_{0}, \mathbf{X})$ instead of maximizing $\ln L(\boldsymbol{\theta}|\mathbf{X})$ # Algorithm 1. Expectation Step: Compute $$Q(\boldsymbol{\theta} | \hat{\boldsymbol{\theta}}^{(m)}, \mathbf{X}) := E_{\hat{\boldsymbol{\theta}}^{(m)}}[\ln L^{c}(\boldsymbol{\theta}|\mathbf{X}, \mathbf{Z}) | \hat{\boldsymbol{\theta}}_{m}, \mathbf{X}]$$ where the $m = 0, 1, \dots$, and the expectation is taken under the conditional pdf $k(\mathbf{Z} | \mathbf{X}, \hat{\boldsymbol{\theta}}^{(m)})$ 2. Maximization Step: $$\hat{\boldsymbol{\theta}}^{(m+1)} = \underset{\boldsymbol{\theta}}{\operatorname{arg max}} Q(\boldsymbol{\theta} | \hat{\boldsymbol{\theta}}^{(m)}, \mathbf{X})$$ # Properties ## Convergence The [[Sequence]] of estimates $\hat{\boldsymbol{\theta}}^{(m)}$ satisfies $$L(\hat{\boldsymbol{\theta}}^{(m+1)}|\mathbf{X}) \leq L(\hat{\boldsymbol{\theta}}^{(m)}|\mathbf{X})$$ Therefore the [[Sequence]] of EM estimates converge to (at least local) optimalLink to original

Sufficiency

Decision Function

Definition

Let $X_1,X_2,\dots ,X_n$ be Random Sample with PDF $f(x|\theta )$, where $\theta \in \Omega$, $X:=u(X_1,X_2,\dots ,X_n)$ be a Statistic for the parameter $\theta$, and $\delta (x)$ be a function of the observed value of the statistic $X$, then The function $\delta$ is called a decision function or a decision rule.

Link to original

Loss Function

Definition

Let $\theta \in \Omega$ be a parameter, $X:=u(X_1,X_2,\dots ,X_n)$ be a Statistic for the parameter $\theta$, and $\delta (x)$ be a Decision Function

A loss function is a non-negative function defined as $L(\theta ,\delta (x))$

It indicates the difference or discrepancy between $\theta$ and $\delta (x)$

Examples

• Absolute Error Loss
• Squared Error Loss
• Sum of Squared Errors Loss
• Cross-Entropy Loss
• Goal Post Error Loss
• Huber Loss
• Binary Loss
• Triplet Loss
• Pairwise Loss

Link to original

Risk Function

Definition

$R(\theta ,\delta )=E[L(\theta ,\delta (x))]={\int }_{X}L(\theta ,\delta (x))f(x|\theta )dx$

Risk function is an expectation of Loss Function

Link to original

Minimum Variance Unbiased Estimator

Definition

An estimator $Y=u(X_1,X_2,\dots ,X_n)$ satisfying the following is the minimum variance unbiased estimator (MVUE) for $\theta$ $(E(Y)=\theta )\wedge (\forall T,(E(T)=\theta )\Rightarrow \mathrm{Var}(Y)\le \mathrm{Var}(T))$ where $T$ is an unbiased estimator

An Unbiased Estimator that has lower variance than any other unbiased estimator for the parameter $\theta$

Facts

A minimum variance unbiased estimator does not always exist.

If some unbiased estimator’s variance is equal to the Rao-Cramer Lower Bound for all $\theta$, then it is a minimum variance unbiased estimator.

Link to original

Minimax Estimator

Definition

An estimator ${\delta }^{\ast }(x)$ satisfying the following is he minimax estimator of $\theta$ $\forall \delta (x),\underset{\theta }{\max }R(\theta ,{\delta }^{\ast }(x))\le \underset{\theta }{\max }R(\theta ,\delta (x))$

Link to original

A Sufficient Statistic for a Parameter

Sufficient Statistic

Definition

Let $X_1,X_2,\dots ,X_n$ be a Random Sample with PDF $f(x|\theta )$, where $\theta \in \Omega$, and $Y_1=u_1(X_1,X_2,\dots ,X_n)$ be a Statistic with PDF $f_{Y_1}(y_1|\theta )$. The $Y_1$ is a sufficient statistic for $\theta$ if and only of $\frac{{\prod }_{i=1}^{n}f(x_i|\theta )}{f_{Y_i}(y_1|\theta )}=H(x_1,x_2,\dots ,x_n)$

No other statistic that can be calculated from the same sample provides any additional information as to the value of the parameter.

Facts

Any one-to-one transformation of a sufficient statistic is also sufficient statistic

MLE is a function of sufficient statistic

Let $X_1,X_2,\dots ,X_n$ be a Random Sample with PDF $f(x|\theta )$, where $\theta \in \Omega$, $Y_1=u_1(X_1,X_2,\dots ,X_n)$ be a Statistic with PDF $f_{Y_1}(y_1|\theta )$, and $\hat{\theta }$ be a unique MLE of $\theta$, then $\hat{\theta }$ is a function of $Y_1$

Link to original

Neyman's Factorization Theorem

Definition

Let $X_1,X_2,\dots ,X_n$ be a Random Sample with PDF $f(x|\theta )$, where $\theta \in \Omega$, then The statistic $Y_1=u_1(X_1,X_2,\dots ,X_n)$ is a Sufficient Statistic for $\theta$ if and only if $\exists k_1,k_2,{\prod }_{i=1}^{n}f(x_i|\theta )=k_1(y_1|\theta )k_2(x_1,x_2,\dots ,x_n)$ where $k_1,k_2$ are non-negative functions, and $k_2$ does not depend on $\theta$

Link to original

Rao-Blackwell Theorem

Definition

Let $X_1,X_2,\dots ,X_n$ be a Random Sample with PDF $f(x|\theta )$, where $\theta \in \Omega$, $Y_1=u_1(X_1,X_2,\dots ,X_n)$ be a Sufficient Statistic for $\theta$, $Y_2=u_2(X_1,X_2,\dots ,X_n)$ be a Unbiased Estimator for $\theta$. And define $\phi (Y_1):=E(Y_2|Y_1)$, then $\mathrm{Var}[\phi (Y_1)]\le \mathrm{Var}(Y_2)\wedge E(\phi (Y_1))=\theta$ $\phi (Y_1)$ is an Unbiased Estimator, and it has a lower variance than $Y_2$

Link to original

Completeness and Uniqueness

Complete Statistic

Definition

Let $Z$ be a Random Variable with PDF $h(z|\theta )$, where $\theta \in \Omega$. A complete statistic $Z$ for $\theta$ satisfies the condition $\forall \theta \in \Omega ,E[u(Z)]=0\Rightarrow u(Z)=0$ then, $\{h(z|\theta )|\theta \in \Omega \}$ is called a complete family.

Link to original

Uniformly Minimum Variance Unbiased Estimator

Definition

$\forall \theta \in \Omega ,(E(Y)=\theta )\wedge (\forall T,(E(T)=\theta )\Rightarrow \mathrm{Var}(Y)\le \mathrm{Var}(T))$ where $T$ is an unbiased estimator, and $\theta$ is the parameter.

An Unbiased Estimator that has lower variance than any other unbiased estimator for all possible values of the parameter $\forall \theta \in \Omega$

An MVUE for all possible values of the parameters

Link to original

Lehmann-Scheffe Theorem

Definition

Let $X_1,X_2,\dots ,X_n$ be a Random Sample with PDF $f(x|\theta )$, where $\theta \in \Omega$, $Y_1=u_1(X_1,X_2,\dots ,X_n)$ be a complete sufficient Statistic for $\theta$, then If $E[\phi (Y_1)]=\theta$, then $\phi (Y_1)$ is the unique UMVUE of $\theta$

Facts

Let $Y_1$ be a complete sufficient Statistic for $\theta$, $Y_2$ be a Unbiased Estimator of $q(\theta )$

• $\exists h,E[h(Y_1)]=q(\theta )$, then $h(Y_1)$ is a unique UMVUE of $q(\theta )$
• By the Rao-Blackwell Theorem, $E(Y_2|Y_1)$ is a unique UMVUE of $q(\theta )$

Link to original

The Exponential Class of Distribution

Regular Exponential Class

Definition

A PDF of form $f(x|\theta )=\exp [p(\theta )K(x)+H(x)+q(\theta )]I(x\in \mathcal{S})$ is said to be a member of the regular exponential class if

• $\mathcal{S}$, support of $x$, does not depend on $\theta$
• $p(\theta )$ is a non-trivial (constant) continuous function of $\theta \in \Omega$
• If $X$ is a continuous random variable, then $K^{\prime }(x)\ne 0$ and $H(x)$ is a continuous function of $x\in \mathcal{S}$
• If $X$ is a discrete random variable, then $K(x)$ is non-trivial function of $x\in \mathcal{S}$

Properties

Joint PDF

Let $X_1,X_2,\dots ,X_n$ be a Random Sample from a regular exponential class, then the joint pdf of $X_1,X_2,\dots ,X_n$ is ${\prod }_{i=1}^{n}f(x_i|\theta )=\exp \left[p(\theta ){\sum }_{i=1}^{n}K(x_i)+{\sum }_{i=1}^{n}H(x_i)+nq(\theta )\right]{\prod }_{i=1}^{n}I(x_i\in \mathcal{S})$

Monotone Likelihood Ratio

Let $X_1,X_2,\dots ,X_n$ be a Random Sample from a regular exponential class $f(x|\theta )=\exp [p(\theta )K(x)+H(x)+q(\theta )]I(x\in \mathcal{S})$ If $p(\theta )$ is monotone function of $\theta$, then likelihood ratio has Monotone Likelihood Ratio property in $y=\sum _{i=1}^{n}K(x_i)$

Facts

Let $X_1,X_2,\dots ,X_n$ be a Random Sample from a regular exponential class, and $Y_1=\sum _{i=1}^{n}K(x_i)$ be a Statistic, then

• The pdf of $Y_1$ is $f_{Y_1}(y_1|\theta )=R(y_1)\exp [p(\theta )y_1+nq(\theta )]$ where $R(y_1)$ is a function of $Y_1$ only.
• $E(Y_1)=-n\frac{q^{\prime }(\theta )}{p^{\prime }(\theta )}$
• $\mathrm{Var}(Y_1)=n\frac{1}{p^{\prime }(\theta {)}^3}\{p^{\prime \prime }(\theta )q^{\prime }(\theta )-q^{\prime \prime }(\theta )p^{\prime }(\theta )\}$
• $Y_1$ is a complete sufficient Statistic

Link to original

Function of a Parameter

Multiparametric Sufficiency

Definition

Joint Sufficient Statistic

Let $X_1,X_2,\dots ,X_n$ be a Random Sample with PDF $f(x|\mathbf{\theta })$, where $\mathbf{\theta }\in \Omega \subset {\mathbb{R}}^p$, $\mathbf{Y}=(Y_1,Y_2,\dots ,Y_m{)}^{⊺}$, where $Y_i=u_i(X_1,X_2,\dots ,X_n)$ be a Statistic, and $f_{\mathbf{Y}}(\mathbf{y}|\mathbf{\theta })$ be a pdf of $\mathbf{y}$. The $\mathbf{Y}$ is jointly sufficient for $\mathbf{\theta }$ if and only $\frac{{\prod }_{i=1}^{n}f(x_i|\mathbf{\theta })}{f_{\mathbf{Y}}(\mathbf{y}|\mathbf{\theta })}=H(x_1,x_2,\dots ,x_n),\forall x_i\in \mathcal{S}$ where $H(x_1,x_2,\dots ,x_n)$ does not depend on $\mathbf{\theta }$

if and only if ${\prod }_{i=1}^{n}f(x_i|\mathbf{\theta })=k_1(\mathbf{y}|\mathbf{\theta })k_2(x_1,x_2,\dots ,x_n),\forall x_i\in \mathcal{S}$ where $k_2(x_1,x_2,\dots ,x_n)$ does not depend on $\mathbf{\theta }$

Completeness

Let $\{f(x_1,v_2,\dots ,v_k|\mathbf{\theta })|\mathbf{\theta }\in \Omega \}$ be a family of pdfs of $k$ random variables $V_1,V_2,\dots ,V_n$ If $\forall \mathbf{\theta }\in \Omega ,E[u(v_1,v_2,\dots ,v_n)]=0\Rightarrow u(v_1,v_2,\dots ,v_n)=0$, then the family of pdfs is called complete family, and $V_1,V_2,\dots ,V_k$ are complete statistics for $\mathbf{\theta }$

m-Dimensioanl Regular Exponential Class

Let $X$ be a Random Variable with PDF $f(x|\mathbf{\theta })$, where $\mathbf{\theta }\in \Omega \subset {\mathbb{R}}^m$, and $\mathcal{S}$ be a support of the $X$. If $f(x|\mathbf{\theta })$ can be written as $f(x|\mathbf{\theta })=\exp \left[\sum _{j=1}^m{p}_j(\mathbf{\theta })K_j(x)+H(x)+q({\theta }_1,{\theta }_2,\dots ,{\theta }_m)\right]I_{\mathcal{S}}(x)$ then, it is called $m$-dimensional exponential class. Further, it is called a regular case if the following are satisfied

• $\mathcal{S}$, support of $x$, does not depend on $\mathbf{\theta }$
• $\Omega$ contains m-dimensional open rectangle
• $p_1(\mathbf{\theta }),p_2(\mathbf{\theta }),p_m(\mathbf{\theta })$ are functionally independent and continuous function of $\mathbf{\theta }$
• If $X$ is a continuous random variable, then $K_j^{\prime }(x)$ and $H(x)$ are continuous function
• If $X$ is a discrete random variable, then $K_j(x)$ are non-trivial function of $x\in \mathcal{S}$

Let $X_1,X_2,\dots ,X_n$ be a Random Sample from a m-dimensional regular exponential class, then the joint pdf of $X_1,X_2,\dots ,X_n$ is ${\prod }_{i=1}^{n}f(x_i|\mathbf{\theta })=\exp \left[{\sum }_{j=1}^m{p}_j(\mathbf{\theta })\left({\sum }_{i=1}^n{K}_j(x_i)\right)+{\sum }_{i=1}^{n}H(x_i)+nq(\mathbf{\theta })\right]{\prod }_{i=1}^{n}I(x_i\in \mathcal{S})$

$\mathbf{Y}=(Y_1,Y_2,\dots ,Y_m{)}^{⊺}$, where $Y_j=\sum _{i=1}^n{K}_j(x_i)$, is a joint complete sufficient Statistic for $\mathbf{\theta }$

the joint pdf of $\mathbf{Y}=(Y_1,Y_2,\dots ,Y_m{)}^{⊺}$ is $f_{\mathbf{Y}}(\mathbf{y}|\mathbf{\theta })=R(\mathbf{y})\exp \left[\sum _{j=1}^m{p}_j(\mathbf{\theta })y_j+nq(\mathbf{\theta })\right]$, where $R(y)$ does not depend on $\mathbf{\theta }$

k-Dimensional Random Vector with p-Dimensional Parameters

Exponential Class

Let $\mathbf{X}$ be a $k$-dimensional random vector with pdf $f(\mathbf{x}|\mathbf{\theta })$, where $\mathbf{\theta }\in \Omega \subset {\mathbb{R}}^p$ $f(\mathbf{x}|\mathbf{\theta })=\exp \left[\sum _{j=1}^m{p}_j(\mathbf{\theta })K_j(\mathbf{x})+H(\mathbf{x})+q(\mathbf{\theta })\right]I_{\mathcal{S}}(\mathbf{x})$

Link to original

Minimum Sufficiency and Ancillary Statistic

Minimal Sufficient Statistic

Definition

A Sufficient Statistic is called minimal sufficient statistic (MSS) if it has the minimum dimension among all sufficient statistics

Facts

complete sufficient Statistic $\underset{\times }{\stackrel{\circ }{\rightleftharpoons }}$ minimal sufficient statistic

Link to original

Ancillary Statistic

Definition

Let $X_1,X_2,\dots ,X_n$ be a Random Sample with PDF $f(x|\theta )$, where $\theta \in \Omega$ $Y_1=u_1(X_1,X_2,\dots ,X_n)$ is called ancillary statistic to $\theta$ if its distribution does not depend on the $\theta$

Link to original

Location Parameter

Definition

Let $X_i=\theta +W_i$, where $W_1,W_2,\dots ,W_n$, be Random Sample from a pdf $f_W(w)$ If $f_{X_i}(x)\stackrel{D}{=}{f}_W(x-\theta )$, then $\theta$ is called location parameter

Link to original

Location Invariant Statistic

Definition

Let $X_1,X_2,\dots ,X_n$ be a Random Sample from a location model, and $Z=u(X_1,X_2,\dots ,X_n)$ be a statistic such that $\forall d\in \mathbb{R},u(x_1+d,x_2+d,\dots ,x_n+d)=u(x_1,x_2,\dots ,x_n)$, then $Z=u(\theta +W_1,\theta +W_2,\dots ,\theta +W_n)=u(W_1,W_2,\dots ,W_n)$. So, the distribution of $Z$ does not depend on $\theta$. In this case, $Z$ is Ancillary Statistic to $\theta$, and it is called location-invariant statistic.

Link to original

Scale Parameter

Definition

Let $X_i=\theta W_i$, where $W_1,W_2,\dots ,W_n$, be Random Sample from a pdf $f_W(w)$ If $f_{X_i}(x)=\frac{1}{\theta }{f}_W\left(\frac{x}{\theta }\right)$, then $\theta$ is called scale parameter

Link to original

Scale-Invariant Statistic

Definition

Let $X_1,X_2,\dots ,X_n$ be a Random Sample from a scale model, and $Z=u(X_1,X_2,\dots ,X_n)$ be a statistic such that $\forall c\in {\mathbb{R}}^{+},u(c{x}_1,c{x}_2,\dots ,c{x}_n)=u(x_1,x_2,\dots ,x_n)$, then $Z=u(\theta W_1,\theta W_2,\dots ,\theta W_n)=u(W_1,W_2,\dots ,W_n)$. So, the distribution of $Z$ does not depend on $\theta$. In this case, $Z$ is Ancillary Statistic to $\theta$, and it is called scale-invariant statistic.

Link to original

Location-Scale Family

Definition

Let $X_i={\theta }_1+{\theta }_2{W}_i$, where $W_1,W_2,\dots ,W_n$, be Random Sample from a pdf $f_W(w)$ If $f_{X_i}(x)\stackrel{D}{=}\frac{1}{{\theta }_2}{f}_W\left(\frac{x-{\theta }_1}{{\theta }_2}\right)$, then ${\theta }_1$ is called location parameter and ${\theta }_2$ is called scale parameter

Link to original

Location- and Scale-Invariant Statistic

Definition

Let $X_1,X_2,\dots ,X_n$ be a Random Sample from a location-scale model, and $Z=u(X_1,X_2,\dots ,X_n)$ be a statistic such that $\forall d\in \mathbb{R},\forall c\in {\mathbb{R}}^{+},u(c{x}_1+d,c{x}_2+d,\dots ,c{x}_n+d)=u(x_1,x_2,\dots ,x_n)$, then $Z=u({\theta }_1+{\theta }_2{W}_1,{\theta }_1+{\theta }_2{W}_2,\dots ,{\theta }_1+{\theta }_2{W}_n)=u(W_1,W_2,\dots ,W_n)$. So, the distribution of $Z$ does not depend on ${\theta }_1,{\theta }_2$. In this case, $Z$ is Ancillary Statistic to $\theta$, and it is called location- and scale-invariant statistic.

Link to original

Sufficiency, Completeness, and Independence

Basu's Theorem

Definition

Let $X_1,X_2,\dots ,X_n$ be a Random Sample with PDF $f(x|\theta )$, where $\theta \in \Omega$, $Y$ be a complete sufficient Statistic for $\theta$, and $Z$ be a Ancillary Statistic to $\theta$, then UMP $Y$ and $Z$ are independent.

Link to original

Optimal Tests of Hypothesis

Most Powerful Test

Best Critical Region

Definition

Let $C$ be a subset of sample space. $C$ is a best critical region of size $\alpha$ for testing the simple hypothesis test $H_0:\theta ={\theta }^{\prime },H_1:\theta ={\theta }^{\prime \prime }$ if

• $P_{{\theta }^{\prime }}[X\in C]=\alpha$ (level alpha condition)
• $\forall A,P_{{\theta }^{\prime }}(X\in \alpha )\Rightarrow P_{{\theta }^{\prime \prime }}(X\in C)\ge P_{{\theta }^{\prime \prime }}(X\in A)$ (most powerful)

Link to original

Most Powerful Test

Definition

A test with the Best Critical Region is called most powerful (MP) test

Link to original

Neyman-Pearson Theorem

Definition

The Neyman-Pearson theorem states that the Most Powerful Test for choosing between two simple hypotheses is based on the likelihood ratio.

Let $X_1,X_2,\dots ,X_n$ be random samples with PDF $f(x|\theta )$, where $\theta \in \Omega =\{{\theta }_0,{\theta }_1\}$, then the Likelihood Function of $X_1,X_2,\dots ,X_n$ is defined as $L(\theta |\mathbf{x})=\prod _{i=1}^{n}f(x_i|\theta )$ Consider the simple null hypothesis $H_0:\theta ={\theta }_0$ against the simple alternative hypothesis $\theta ={\theta }_1$ and define the likelihood ratio test statistic. $\Lambda (\mathbf{x})=\frac{L({\theta }_1|\mathbf{x})}{L({\theta }_0|\mathbf{x})}$

Let $k\in {\mathbb{R}}^{+}$, and $C$ be a subset of the sample space such that

• $\forall \mathbf{x}\in C^{\complement },\Lambda (\mathbf{x})\le k$
• $\forall \mathbf{x}\in C,\Lambda (\mathbf{x})\ge k$
• $P_{{\theta }_0}(\mathbf{X}\in C)=\alpha$ where ${\mathbf{x}}^{⊺}=(x_1,x_2,\dots ,x_n)$

Then, $C$ is a Best Critical Region of size $\alpha$ for the hypothesis test.

Link to original

Uniformly Most Powerful Tests

Uniformly Most Powerful Critical Region

Definition

If $C$ is a Best Critical Region of size $\alpha$ for every $H_0$ in $H_1$, then $C$ is called a uniformly most powerful critical region

Link to original

Uniformly Most Powerful Test

Definition

A test defined by Uniformly Most Powerful Critical Region is called a uniformly most powerful (UMP) test

Simple vs Composite

Consider $H_0:\theta ={\theta }^{\prime }$ is a simple hypothesis and $H_1:\theta >{\theta }^{\prime },(\theta \ne {\theta }^{\prime },\theta <{\theta }^{\prime })$ is a composite hypothesis. Fix some ${\theta }^{\prime \prime }$ satisfying $H_1$ and obtain a most powerful test using Neyman-Pearson Theorem and generalize it to all ${\theta }^{\prime \prime }$ satisfying $H_1$ conditions

Composite vs Composite

Karlin-Rubin Theorem

Definition

Consider the hypothesis $H_0:\theta \le {\theta }^{\prime },(\theta \ge {\theta }^{\prime })$ and $H_1:\theta >{\theta }^{\prime }(\theta <{\theta }^{\prime })$, and assume that the likelihood ratio has MLR property in some Sufficient Statistic $Y=u(\mathbf{x})$ If the likelihood ratio is decreasing (increasing) in the statistic $Y=u(\mathbf{x})$ Then the Uniformly Most Powerful Test of level $\alpha$ for the hypothesis is rejecting $H_0$ if $Y\ge c,(Y\le c)$, where $c$ is determined by $\alpha =P_{{\theta }^{\prime }}(Y\ge c)$

Link to original

Link to original

Monotone Likelihood Ratio

Definition

$L(\theta |\mathbf{x})$ has a monotone likelihood ratio (MLR) property in $y=u(\mathbf{x})$ If $\forall {\theta }_1<{\theta }_2,\frac{L({\theta }_1|\mathbf{x})}{L({\theta }_2|\mathbf{x})}$ is a monotone function of $y=u(\mathbf{x})$

Link to original

Karlin-Rubin Theorem

Definition

Consider the hypothesis $H_0:\theta \le {\theta }^{\prime },(\theta \ge {\theta }^{\prime })$ and $H_1:\theta >{\theta }^{\prime }(\theta <{\theta }^{\prime })$, and assume that the likelihood ratio has MLR property in some Sufficient Statistic $Y=u(\mathbf{x})$ If the likelihood ratio is decreasing (increasing) in the statistic $Y=u(\mathbf{x})$ Then the Uniformly Most Powerful Test of level $\alpha$ for the hypothesis is rejecting $H_0$ if $Y\ge c,(Y\le c)$, where $c$ is determined by $\alpha =P_{{\theta }^{\prime }}(Y\ge c)$

Link to original

The Sequential Probability Ratio Test

Sequential Probability Ratio Test

Definition

Let $X_1,X_2,\dots ,X_n$ be Random Sample with PDF $f(x|\mathbf{\theta })$, where $\theta \in \Omega =\{{\theta }^{\prime },{\theta }^{\prime \prime }\}$, and $L(\theta |n)={\prod }_{i=1}^{n}f(x_i|\theta )$ be the Likelihood Function Consider a simple test $H_0:\theta ={\theta }^{\prime },H_1:\theta ={\theta }^{\prime \prime }$ We observe likelihood ratio sequentially i.e. $\frac{L({\theta }^{\prime }|1)}{L({\theta }^{\prime \prime }|1)},\frac{L({\theta }^{\prime }|2)}{L({\theta }^{\prime \prime }|2)},\dots$, and reject $H_0$ if and only if $\mathbf{x}=(x_1,x_2,\dots ,x_n)\in C_n$, where $C_n=\left\{(\mathbf{x}|\forall j=\{1,2,\dots ,n-1\},k_0<\frac{L({\theta }^{\prime }|j)}{L({\theta }^{\prime \prime }|j)}<k_1\wedge \frac{L({\theta }^{\prime }|n)}{L({\theta }^{\prime \prime }|n)}\le k_0\right\}$, and do not reject $H_0$ if and only if $\mathbf{x}=(x_1,x_2,\dots ,x_n)\in B_n$, where $B_n=\left\{(\mathbf{x}|\forall j=\{1,2,\dots ,n-1\},k_0<\frac{L({\theta }^{\prime }|j)}{L({\theta }^{\prime \prime }|j)}<k_1\wedge \frac{L({\theta }^{\prime }|n)}{L({\theta }^{\prime \prime }|n)}\ge k_1\right\}$

i.e. we continue to observe as long as $k_0<\frac{L({\theta }^{\prime }|j)}{L({\theta }^{\prime \prime }|j)}<k_1$, and stop otherwise.

Choice of $k_0$ and $k_1$

Let $\alpha ,\beta$ be type 1 and 2 error, respectively, then $\alpha =\sum _{i=1}^{\infty }{\int }_{C_n}L({\theta }^{\prime }|n),\beta =\sum _{i=1}^{\infty }{\int }_{B_n}L({\theta }^{\prime \prime }|n),1-\alpha =\sum _{i=1}^{\infty }{\int }_{B_n}L({\theta }^{\prime }|n),1-\beta =\sum _{i=1}^{\infty }{\int }_{C_n}L({\theta }^{\prime \prime }|n)$ Hence, $\alpha =\sum _{i=1}^{\infty }{\int }_{C_n}L({\theta }^{\prime }|n)\le \sum _{i=1}^{\infty }{\int }_{C_n}{k}_{0}L({\theta }^{\prime \prime }|n)=k_0(1-\beta )$ $1-\alpha =\sum _{i=1}^{\infty }{\int }_{B_n}L({\theta }^{\prime }|n)\ge \sum _{i=1}^{\infty }{\int }_{B_n}{k}_{1}L({\theta }^{\prime \prime }|n)=k_1\beta$ Therefore, $\frac{\alpha }{1-\beta }<k_0<k_1<\frac{1-\alpha }{\beta }$ By the inequality, we choose $k_0:=\frac{\alpha }{1-\beta },k_1:=\frac{1-\alpha }{\beta }$

Link to original

Minimax and Classification Procedures

Minimax Procedures

Definition

We want to test $H_0:\theta ={\theta }^{\prime },H_1:\theta ={\theta }^{\prime \prime }$ Let $\mathcal{L}(\theta ,\delta )$ be a Loss Function such that $\mathcal{L}({\theta }^{\prime },{\theta }^{\prime })=\mathcal{L}({\theta }^{\prime \prime },{\theta }^{\prime \prime })=0$ and $\mathcal{L}({\theta }^{\prime },{\theta }^{\prime \prime })>0,\mathcal{L}({\theta }^{\prime \prime },{\theta }^{\prime })>0$, $C$ be a rejection region, $C^{\complement }$ be an acceptance region, and $R(\theta ,\delta )=E[\mathcal{L}(\theta ,\delta )]$ be a Risk Function Then, $R({\theta }^{\prime },C)=\alpha \mathcal{L}({\theta }^{\prime },{\theta }^{\prime \prime }),R({\theta }^{\prime \prime },C)=\beta \mathcal{L}({\theta }^{\prime \prime },{\theta }^{\prime })$ The minimax procedure is to find $C$ such that $\max \{R({\theta }^{\prime },C),R({\theta }^{\prime \prime },C)\}$ is minimized

The minimax solution is the region $C=\left\{(x_1,x_2,\dots ,x_n)|\frac{L({\theta }^{\prime })}{L({\theta }^{\prime \prime })}\le k\right\}$, where $k$ satisfies $R({\theta }^{\prime },C)=R({\theta }^{\prime \prime },C)$

Link to original

Inferences About Normal Linear Models

Quadratic Form

Definition

$q(\mathbf{x})={\mathbf{x}}^{⊺}\mathbf{Ax}$ where $A$ is a Symmetric Matrix

A mapping $Q:V\to K$ where $V$ is a Module on Commutative Ring $K$ that has the following properties. $\forall k,l\in K,\forall u,v,w\in V$

• $Q(kv)=k^{2}Q(v)$
• $Q(u+v+w)=Q(u+v)+Q(v+w)+Q(u+w)-Q(u)-Q(v)-Q(w)$
• $Q(ku+lv)=k^{2}Q(u)+l^{2}Q(v)+klQ(u+v)-klQ(u)-klQ(v)$

Matrix Expressions

$a_1{x}_1^2+a_2{x}_2^2+2a_3{x}_1{x}_2\iff \left[\begin{array}{cc}{x}_1& x_2\end{array}\right]\left[\begin{array}{cc}{a}_1& a_3\\ a_3& a_2\end{array}\right]\left[\begin{array}{c}{x}_1\\ x_2\end{array}\right]$

$a_1{x}_1^2+a_2{x}_2^2+a_3{x}_3^2+2a_4{x}_1{x}_2+2a_5{x}_1{x}_3+2a_6{x}_2{x}_3\iff \left[\begin{array}{ccc}{x}_1& x_2& x_3\end{array}\right]\left[\begin{array}{ccc}{a}_1& a_4& a_5\\ a_4& a_2& a_6\\ a_5& a_6& a_3\end{array}\right]\left[\begin{array}{c}{x}_1\\ x_2\\ x_3\end{array}\right]$

Facts

Let $\mathbf{x}$ be a Random Vector and $A$ be a symmetric matrix of constants. If $E(\mathbf{x})=\mathbf{\mu }$ and $\mathrm{Var}(\mathbf{x})=\Sigma$, then the expectation of the quadratic form $Q:={\mathbf{x}}^{⊺}A\mathbf{x}$ is $E(Q)={\mathbf{\mu }}^{⊺}A\mathbf{\mu }+\mathrm{tr}(A\Sigma )$

Let $\mathbf{x}=(X_1,\dots ,X_n{)}^{⊺}$ be a Random Vector and $A$ be a symmetric matrix of constants. If $E(\mathbf{x})=\mathbf{\theta }=({\theta }_1,\dots ,{\theta }_n{)}^{⊺}$ and $\mathrm{Var}(X_i)={\mu }_2$, $E[(X_i-{\theta }_i{)}^3]={\mu }_3$, and $E[(X_i-{\theta }_i{)}^4]={\mu }_4$, where $i=1,2,\dots ,n$, then the variance of the quadratic form ${\mathbf{x}}^{⊺}A\mathbf{x}$ is $\mathrm{Var}({\mathbf{x}}^{⊺}A\mathbf{x})=({\mu }_4-3{\mu }_2^2){\mathbf{a}}^{⊺}\mathbf{a}+2{\mu }_2^2\mathrm{tr}(A^2)+4{\mu }_2{\mathbf{\theta }}^{⊺}{A}^2\mathbf{\theta }+4{\mu }_3{\mathbf{\theta }}^{⊺}A\mathbf{a}$ where $\mathbf{a}$ is the column vector of diagonal elements of $A$.

If $X_i\sim N({\theta }_i,{\mu }_2),i=1,\dots ,n$ and $X_i$‘s are independent, then $\mathrm{Var}({\mathbf{x}}^{⊺}A\mathbf{x})=2{\mu }_2^2\mathrm{tr}(A^2)+4{\mu }_2{\mathbf{\theta }}^{⊺}{A}^2\mathbf{\theta }$ If $X_i\sim N(0,1),i=1,\dots ,n$ and $X_i$‘s are independent, then $\mathrm{Var}({\mathbf{x}}^{⊺}A\mathbf{x})=2\mathrm{tr}(A^2)$

Let ${\mathbf{X}}^{⊺}\sim N(0,{\sigma }^2{I}_n)$, $Q:={\mathbf{X}}^{⊺}A\mathbf{X}/{\sigma }^2$, where $A$ is a Symmetric Matrix and $\mathrm{rank}(A)=r\le n$, then the MGF of $Q$ is $M(t)={\prod }_{i=1}^r(1-2t{\lambda }_i{)}^{-1/2}=|I-2tA{|}^{-1/2}$ where ${\lambda }_i$‘s are non-zero eigenvalue of $A$

Let $\mathbf{X}\sim N_n(\mathbf{\mu },\Sigma )$, where $\Sigma$ is Positive-Definite Matrix, then $Q=(\mathbf{X}-\mathbf{\mu }{)}^{⊺}{\Sigma }^{-1}(\mathbf{X}-\mathbf{\mu })\sim {\chi }^2(n)$

Let $\mathbf{X}\sim N(0,{\sigma }^2{I}_n)$, $Q={\mathbf{X}}^{⊺}A\mathbf{X}/{\sigma }^2$, where $A$ is a Symmetric Matrix and $\mathrm{rank}(A)=r\le n$, then $Q\sim {\chi }^2(r)\iff A=A^k$ where $k\in \mathbb{N}$

Let $\mathbf{X}\sim N(0,{\sigma }^2{I}_n)$, $Q_1={\mathbf{X}}^{⊺}A\mathbf{X}/{\sigma }^2,Q_2={\mathbf{X}}^{⊺}B\mathbf{X}/{\sigma }^2$, where $A,B$ are symmetric matrices, then $Q_1,Q_2$ are independent if and only if $AB=O$

Let $Q=Q_1+Q_2+\cdots +Q_{k-1}+Q_k$, where $Q,Q_1,Q_2,\dots ,Q_k$ are quadratic forms in Random Sample from $N(0,{\sigma }^2)$ If $Q/{\sigma }^2\sim {\chi }^2(r),Q_1/{\sigma }^2\sim {\chi }^2(r_1),Q_2/{\sigma }^2\sim {\chi }^2(r_2),\dots ,Q_{k-1}/{\sigma }^2\sim {\chi }^2(r_{k-1})$ and $Q_k$ is non-negative, then

• $Q_1,Q_2,\dots ,Q_k$ are independent
• $Q_k/{\sigma }^2\sim {\chi }^2(r-r_1-r_2-\cdots -r_{k-1})$

Let $\mathbf{X}=(X_1,X_2,\dots ,X_n)\sim N(0,{\sigma }^2{I}_n)$, $\sum _{i=1}^n{X}_i^2=Q_1+Q_2+\cdots +Q_k$, where $Q_j={\mathbf{X}}^{⊺}{A}_j\mathbf{X}$, where $\mathrm{rank}(A_j)=r_j\le n$, then $\forall j=\{1,2,\dots ,k\},\frac{Q_j}{{\sigma }^2}\sim {\chi }^2(r_j)\iff {\sum }_{j=1}^k{r}_j=n$

${\mathbf{x}}^{⊺}A\mathbf{x}-2{\mathbf{x}}^{⊺}\mathbf{b}=(\mathbf{x}-A^{-1}\mathbf{b}{)}^{⊺}A(\mathbf{x}-A^{-1}\mathbf{b})-{\mathbf{b}}^{⊺}{A}^{-1}\mathbf{b}$

Link to original

One-way ANOVA

One-way ANOVA

One-way ANOVA is used to analyze the significance of differences of means between groups. Let an $i$-th response of the $j$-th group be $X_{ij}={\mu }_j+{ϵ}_{ij}$ where ${ϵ}_{ij}\sim N(0,{\sigma }^2),\forall i=1,2,\dots ,n_j,\forall j=1,2,\dots ,c$

We want to test the null hypothesis $H_0:{\mu }_1={\mu }_2=\cdots ={\mu }_c$ i.e. there is no treatment effect. $F=\frac{\sum _{j=1}^c{n}_j({\bar{X}}_{.j}-{\bar{X}}_{..}{)}^2/(c-1)}{\sum _{j=1}^c\sum _{i=1}^{n_j}(X_{ij}-{\bar{X}}_{.j}{)}^2/(N-c)}=\frac{SSB/(c-1)}{SSW/(N-c)}\sim F(c-1,N-c)$ where $N=\sum _{j=1}^c{n}_j$ is the total number of observations, ${\bar{X}}_{.j}$ represents the mean of the $j$-th group and ${\bar{X}}_{..}$ represents the overall mean, $\frac{1}{{\sigma }^2}\sum _{j=1}^c{n}_j({\bar{X}}_{.j}-{\bar{X}}_{..}{)}^2\sim {\chi }^2(c-1)$ of the numerator indicates between-group variance (SSB) and $\frac{1}{{\sigma }^2}\sum _{j=1}^c\sum _{i=1}^{n_j}(X_{ij}-{\bar{X}}_{.j}{)}^2\sim {\chi }^2(N-c)$ of the denominator indicates within-group variance (SSW)

The Likelihood Ratio Test rejects $H_0$ if $F>F_{\alpha }(c-1,N-c)$

Link to original

Noncentral Chi-squared and F distributions

Noncentral Chi-squared Distribution

Definition

Let $X_1,X_2,\dots ,X_k$ be $k$ independent normal distributions $N({\mu }_i,{\sigma }^2)$, then $Y:=\frac{{\sum }_{i=1}^k{X}_i^2}{{\sigma }^2}\sim {\chi }^2(k,\lambda )$ where $k\in \mathbb{N}$ is degree of freedom, and $\lambda :=\sum _{i=1}^k\frac{{\mu }_i^2}{{\sigma }^2}\in {\mathbb{R}}^{+}$ is non-centrality parameter.

Properties

PDF

MGF

$\frac{1}{(1-2t{)}^{k/2}}\exp \left(\frac{\lambda t}{1-2t}\right)$

Link to original

Noncentral F-Distribution

Definition

Let $U\sim {\chi }^2(r_1,\lambda ),V\sim {\chi }^2(r_2)$ be independent random variables following Noncentral Chi-squared Distribution and Chi-squared Distribution, respectively, then $\frac{U/r_1}{V/r_2}\sim F(r_1,r_2,\lambda )$ where $r_1,r_2\in \mathbb{N}$ are degree of freedoms and $\lambda \in {\mathbb{R}}^{+}$ is non-centrality parameter.

Properties

PDF

Link to original

Multiple Comparisons

Multiple Comparisons

Definition

Assume that $H_0:{\mu }_1={\mu }_2=\cdots ={\mu }_K$ is rejected, and we interested in the confidence interval for some linear combination of parameters

Let $X_{1j},X_{2j},\dots ,X_{aj}$‘s be random samples from $N({\mu }_j,{\sigma }^2),j=1,2,\dots ,b$ We want to compute the confidence interval of $\sum _{j=1}^b{k}_j{\mu }_j$, where $k_j\in \mathbb{R}$

Type 1 Error

Consider $m$ multiple testing problem $H_{0i}:{\mu }_{1i}={\mu }_{2i},i=1,2,\dots ,m$, and let $E_i$ be an event of rejecting $H_{0i}$ even though $H_{0i}$ is true (type 1 error). If a test satisfies $P(E_i)\le \alpha$ for level $\alpha$, then the test is called to be controlled at $\alpha$ Under the independence of each test, the probability of type 1 error in $m$ multiple testing is $P(\bigcup _{i=1}^m{E}_i)=1-P(\bigcap _{i=1}^m{E}_i^{\complement })=1-{\prod }_{i=1}^{m}P(E_i^{\complement })\le 1-(1-\alpha {)}^m$ Thus, the probability of type 1 error approaches to $1$ as $m$ increase.

Fixed Constants Case

The $k_j$‘s are fixed constants. The $100(1-\alpha )\%$ confidence interval for each $\sum _{j=1}^b{k}_j{\mu }_j$ is $\sum _{j=1}^b{k}_j{\overline{X}}_{.j}\pm t_{\alpha /2}(b(a-1))\sqrt{\left({\sum }_{j=1}^b{k}_j^2\right)\frac{V}{a}}$ where $V=\frac{1}{b(a-1)}\sum _{i=1}^a\sum _{j=1}^b(X_{ij}-{\overline{X}}_{.j}{)}^2$

Scheffe’s Simultaneous Confidence Interval

The $k_j$‘s are allowed to have any real numbers. The simultaneous $100(1-\alpha )\%$ confidence interval for all $\sum _{j=1}^b{k}_j{\mu }_j$ is $\sum _{j=1}^b{k}_j{\overline{X}}_{.j}\pm \sqrt{\frac{b}{a}V{F}_{\alpha }\left(\sum _{j=1}^b{k}_j^2\right)}$ where $V=\frac{1}{b(a-1)}\sum _{i=1}^a\sum _{j=1}^b(X_{ij}-{\overline{X}}_{.j}{)}^2$

Link to original

The Analysis of Variance

Two-way ANOVA

Definition

Two-way ANOVA Without Replications

	$1$	$2$	$\dots$	$b$
$1$	$X_{11}$	$X_{12}$	$\dots$	$X_{1b}$
$2$	$X_{21}$	$X_{22}$	$\dots$	$X_{2b}$
$⋮$	$⋮$	$⋮$	$\ddots$	$⋮$
$a$	$X_{a1}$	$X_{a2}$	$\dots$	$X_{ab}$

Let $X_{ij}$‘s be Random Sample from $N({\mu }_{ij},{\sigma }^2)$. The ${\mu }_{ij}=\mu +{\alpha }_i+{\beta }_j$ can be decomposed as sum of means (global mean $+$ $i$-th level effect of factor $A$ $+$ $j$-th level effect of factor $B$). In this setup, we assume that $\sum _{i=1}^a{\alpha }_i=\sum _{j=1}^b{\beta }_j=0$

We want to test $H_0:{\beta }_1,{\beta }_2=\cdots ={\beta }_b$. $F=\frac{\sum _{i=1}^a\sum _{j=1}^b({\overline{X}}_{.j}-{\overline{X}}_{..}{)}^2/(b-1)}{\sum _{i=1}^a\sum _{j=1}^b(X_{ij}-{\overline{X}}_{i.}-{\overline{X}}_{.j}+{\overline{X}}_{..}{)}^2/(a-1)(b-1)}\sim F((b-1),(a-1)(b-1))$ where $\sum _{i=1}^a\sum _{j=1}^b({\overline{X}}_{.j}-{\overline{X}}_{..}{)}^2\sim {\chi }^2(b-1)$, and $\sum _{i=1}^a\sum _{j=1}^b(X_{ij}-{\overline{X}}_{i.}-{\overline{X}}_{.j}+{\overline{X}}_{..}{)}^2\sim {\chi }^2((a-1)(b-1))$

The Likelihood Ratio Test rejects $H_0$ if $F>F_{\alpha }((b-1),(a-1)(b-1))$

Under $H_1$, the $F$ follows Noncentral F-Distribution $F^{\prime }=\frac{a\sum _{j=1}^b({\overline{X}}_{.j}-{\overline{X}}_{..}{)}^2/(b-1)}{\sum _{i=1}^a\sum _{j=1}^b(X_{ij}-{\overline{X}}_{i.}-{\overline{X}}_{.j}+{\overline{X}}_{..}{)}^2/(a-1)(b-1)}\sim F\left((b-1),(a-1)(b-1),a{\sum }_{j=1}^b{\beta }_j^2/{\sigma }^2\right)$ where $\frac{1}{{\sigma }^2}a\sum _{j=1}^b({\overline{X}}_{.j}-{\overline{X}}_{..}{)}^2\sim {\chi }^2(b-1,a\sum _{j=1}^b{\beta }_j^2/{\sigma }^2)$

Therefore, the power of the test is $P(F^{\prime }>F_{\alpha })$

Two-way ANOVA With Equal Replications

	$1$	$2$	$\dots$	$b$
$1$	$X_{111},\dots ,X_{11n}$	$X_{121},\dots ,X_{12n}$	$\dots$	$X_{1b1},\dots ,X_{1bn}$
$2$	$X_{211},\dots ,X_{21n}$	$X_{221},\dots ,X_{22n}$	$\dots$	$X_{2b1},\dots ,X_{1bn}$
$⋮$	$⋮$	$⋮$	$\ddots$	$⋮$
$a$	$X_{a11},\dots ,X_{a1n}$	$X_{a21},\dots ,X_{a2n}$	$\dots$	$X_{ab1},\dots ,X_{abn}$

Let $X_{ijk}$‘s be Random Sample from $N({\mu }_{ij},{\sigma }^2)$. The ${\mu }_{ijk}=\mu +{\alpha }_i+{\beta }_j+{\gamma }_{ij}$ can be decomposed as sum of means (global mean + $i$-th level effect of factor $A$ + $j$-th level effect of factor $B$ + interaction effect of $i$-th level of factor $A$ and the $j$-th level of factor $B$ ). In this setup, we assume that $\sum _{i=1}^a{\alpha }_i=\sum _{j=1}^b{\beta }_j=\sum _{i=1}^a{\gamma }_{ij}=\sum _{j=1}^b{\gamma }_{ij}=0$

We want to test $H_0:{\gamma }_{ij}=0,\forall i,j$. $F=\frac{\sum _{i=1}^a\sum _{j=1}^b\sum _{k=1}^n({\overline{X}}_{ij.}-{\overline{X}}_{i..}-{\overline{X}}_{.j.}+{\overline{X}}_{...}{)}^2/(a-1)(b-1)}{\sum _{i=1}^a\sum _{j=1}^b\sum _{k=1}^n(X_{ijk}-{\overline{X}}_{ij.}{)}^2/(N-ab)}\sim F((a-1)(b-1),N-ab)$ where $N=\sum _{i=1}^a\sum _{j=1}^{b}n=abn$, $\sum _{i=1}^a\sum _{j=1}^b\sum _{k=1}^n({\overline{X}}_{ij.}-{\overline{X}}_{i..}-{\overline{X}}_{.j.}+{\overline{X}}_{...}{)}^2\sim {\chi }^2((a-1)(b-1))$, and $\sum _{i=1}^a\sum _{j=1}^b\sum _{k=1}^n(X_{ijk}-{\overline{X}}_{ij.}{)}^2\sim {\chi }^2((N-ab))$

The Likelihood Ratio Test rejects $H_0$ if $F>F_{\alpha }((a-1)(b-1),(N-ab))$

Under $H_1$, the $F$ follows Noncentral F-Distribution $F^{\prime }=\frac{n\sum _{i=1}^a\sum _{j=1}^b({\overline{X}}_{ij.}-{\overline{X}}_{i..}-{\overline{X}}_{.j.}+{\overline{X}}_{...}{)}^2/(a-1)(b-1)}{\sum _{i=1}^a\sum _{j=1}^b\sum _{k=1}^n(X_{ijk}-{\overline{X}}_{ij.}{)}^2/(N-ab)}\sim F\left((a-1)(b-1),(N-ab),n\sum _{i=1}^a\sum _{j=1}^b{\gamma }_{ij}^2/{\sigma }^2\right)$ where $n\sum _{i=1}^a\sum _{j=1}^b({\overline{X}}_{ij.}-{\overline{X}}_{i..}-{\overline{X}}_{.j.}+{\overline{X}}_{...}{)}^2\sim {\chi }^2((a-1)(b-1),n\sum _{i=1}^a\sum _{j=1}^b{\gamma }_{ij}^2/{\sigma }^2)$

Therefore, the power of the test is $P(F^{\prime }>F_{\alpha })$

If $H_0:{\gamma }_{ij}=0,\forall i,j$ is not rejected, then we continue to test $H_0:{\alpha }_1={\alpha }_2=\cdots ={\alpha }_a=0$ or $H_0:{\beta }_1={\beta }_2=\cdots ={\beta }_b=0$

Two-way ANOVA with a Regression Model

$Y={\beta }_0+\sum _{i=1}^{a-1}{\alpha }_i{D}_i+\sum _{j=1}^{b-1}{\beta }_j{E}_j+\sum _{i=1}^{a-1}\sum _{j=1}^{b-1}{\gamma }_{ij}{D}_i{E}_j+ϵ$ where $D_i$ and $E_j$ are dummy variables representing categories of the two factors, $a$ is the number of categories for the first factor, and $b$ is the number of categories for the second factor

In this setting, a corner-point constraint is used for both factors.

The null hypotheses can be tested by Deviance

Deviance Test for Two-way ANOVA

For a two-way ANOVA with factors $A$ and $B$, we have three null hypotheses to test:

• $H_{0A}:{\alpha }_1={\alpha }_2={\alpha }_{a-1}=0$ i.e. there is no treatment effect of factor $A$
• $H_{0B}:{\beta }_1={\beta }_2={\beta }_{b-1}=0$ i.e. there is no treatment effect of factor $B$
• $H_{0AB}:{\gamma }_{ij}=0,\forall i,j$ i.e. there is no interaction effect between factor $A$ and $B$ They can be tested with the Deviance.

If ${\sigma }^2$ is known, the test statistic for $H_{0A}$ is defined as $\Delta D_A=D_0-D_1=\frac{1}{{\sigma }^2}\left(\frac{1}{bn}\sum _{i=1}^a{y}_{i..}^2-\frac{1}{N}{Y}_{...}^2\right)\sim {\chi }^2(a-1)$ And reject the $H_{0A}$ if $\Delta D_A>{\chi }^2(a-1)$

If ${\sigma }^2$ is unknown, the test statistic for $H_{0A}$ is defined as $F_A=\frac{D_0-D_1}{a-1}/\frac{D_1}{N-ab}\sim F(a-1,N-ab)$ And reject the $H_{0A}$ if $F_A>F(a-1,N-ab)$

If ${\sigma }^2$ is known, the test statistic for $H_{0B}$ is defined as $\Delta D_B=D_0-D_1=\frac{1}{{\sigma }^2}\left(\frac{1}{an}\sum _{j=1}^b{y}_{.j.}^2-\frac{1}{N}{Y}_{...}^2\right)\sim {\chi }^2(b-1)$ And reject the $H_{0B}$ if $\Delta D_B>{\chi }^2(b-1)$

If ${\sigma }^2$ is unknown, the test statistic for $H_{0B}$ is defined as $F_B=\frac{D_0-D_1}{b-1}/\frac{D_1}{N-ab}\sim F(b-1,N-ab)$ And reject the $H_{0B}$ if $F_B>F(b-1,N-ab)$

If ${\sigma }^2$ is known, the test statistic for $H_{0AB}$ is defined as $\Delta D_{AB}=D_0-D_1=\frac{1}{{\sigma }^2}\left(\frac{1}{n}\sum _{i=1}^a\sum _{j=1}^b{y}_{ij.}^2-\frac{1}{bn}\sum _{i=1}^a{Y}_{i..}^2-\frac{1}{an}\sum _{j=1}^b{Y}_{.j.}^2+\frac{1}{N}{Y}_{...}^2\right)\sim {\chi }^2((a-1)(b-1))$ And reject the $H_{0AB}$ if $\Delta D_{AB}>{\chi }^2((a-1)(b-1))$

If ${\sigma }^2$ is unknown, the test statistic for $H_{0AB}$ is defined as $F_{AB}=\frac{D_0-D_1}{(a-1)(b-1)}/\frac{D_1}{N-ab}\sim F((a-1)(b-1),N-ab)$ And reject the $H_{0AB}$ if $F_{AB}>F((a-1)(b-1),N-ab)$

Link to original

Nonparametric Statistics

Functional (Statistics)

Definition

$T$ is called a functional if $T$ is a function of CDF or PDF

Let $X_1,X_2,\dots ,X_n$ be Random Sample from a cdf $F(x)$ and $T(F)$ be functional, then the empirical Distribution Function of $F$ at $x$ is ${\hat{F}}_n(x)=\frac{1}{n}\sum _{i=1}^{n}I(X_i\le x)$, and $T({\hat{F}}_n)$ is called an induced estimator of $T(F)$.

Link to original

Location Model

Definition

Location Functional

Let $X$ be continuous Random Sample with CDF $F_X$. If $T(F_X)$ satisfies the following, then it is called location functional

• $\forall a\in \mathbb{R},Y=X+a\Rightarrow T(F_Y)=T(F_X)+a$
• $\forall a\ne 0,Y=aX\Rightarrow T(F_Y)=aT(F_X)$

Location Model

Let ${\theta }_X=T(F_X)$ be a location functional, then $X_1,X_2,\dots ,X_n$ is called to follow a location model with ${\theta }_X=T(F_X)$ if

• $X_i={\theta }_X+{ϵ}_i$, where ${ϵ}_i$‘s are i.i.d. with PDF $f(x)$ and $T(F_{ϵ})=0$

Facts

Mean functional is a locational functional

Let $X$ be Random Variable with CDF $F_X$ and PDF $f_x$ which is symmetric about $a$. If $T(F_X)$ is a location functional, then $T(F_X)=a$

Link to original

Sample Median and Sign Test

Sign Test

Definition

Let $X_i=\theta +{ϵ}_i$, where ${ϵ}_i$‘s are i.i.d. with CDF $F_X$ and PDF $f_x$ with median $0$. i.e. median of $X_i$‘s are $\theta$.

We want to test $H_0:\theta ={\theta }_0,H_1:\theta >{\theta }_0$, which is called sign test Consider a statistic $S({\theta }_0)=\sum _{i=1}^{n}I(X_i>{\theta }_0)$, which is called sign statistic We reject $H_0$ if $S({\theta }_0)\ge c$.

Distribution of Sign Statistic

$I(X_i>{\theta }_0)\sim B(1,\frac{1}{2})$ under $H_0$. Therefore, $S({\theta }_0)\sim B(n,\frac{1}{2})$. Also, by CLT, $\frac{S({\theta }_0)-\frac{n}{2}}{\sqrt{n}/2}\sim N(0,1)$

Consider a sequence of local alternatives $H_0:\theta =0,H_1:{\theta }_n=\delta /\sqrt{n},\delta >0$ The efficacy of the sign statistic $S({\theta }_0)$ is $c_S={\tau }_S^{-1}=2f(0)$

Composite Sign Test

Consider a composite hypothesis $H_0:\theta \le {\theta }_0,H_1:\theta >{\theta }_0$ Since the power function $\gamma (\theta )$ is non-decreasing, the sign test of size $\alpha$ is $\underset{\theta \le {\theta }_0}{\max }\gamma (\theta )$

Facts

Consider a test $H_0:\theta ={\theta }_0,H_1:\theta >{\theta }_0$ and the test statistic $S({\theta }_0)=\sum _{i=1}^{n}I(X_i>{\theta }_0)$, then $\forall k,P_{\theta }(S(0)\ge k)=P_0(S(-\theta )>k)$ Also, the power function $\gamma (\theta )$ of the test is a non-decreasing function of $\theta$

Consider a sequence of hypothesis $H_0:\theta =0,H_1:{\theta }_n=\delta /\sqrt{n},\delta >0$, where $H_1$ is called a local alternative, then ${\lim }_{n\to \infty }\gamma ({\theta }_n)=1-\Phi (z_a-\delta {\tau }_S^{-1})$ where $\Phi$ is a CDF of standard normal distribution, and ${\tau }_S=c_S^{-1}=\frac{1}{2f(0)}$

Let $X_i=\theta +{ϵ}_i$ be random variables from location model, where ${ϵ}_i$‘s are i.i.d. with CDF $F(x)$, PDF $f(x)$, and median 0, and $Q_2$ be the sample median of $X_i$, then $\sqrt{n}(Q_2-\theta )\stackrel{D}{\to }N(0,{\tau }_s^2)$ where ${\tau }_s=\frac{1}{2f(0)}$

Link to original

Efficacy of Sign Test

Definition

Let $T$ be a test statistic and $\mu ({\theta }_n):=E_{{\theta }_n}(T)$, then the efficacy of $T$ is defined as $c_T:={\lim }_{n\to \infty }\frac{{\mu }^{\prime }(0)}{[n\mathrm{Var}(T){]}^{1/2}}$ where ${\mu }^{\prime }(0)$ can be interpreted as the rate of change of the mean of $T$

Link to original

Asymptotic Relative Efficiency

Definition

Let $T_1,T_2$ be two test statistics, then the asymptotic relative efficiency of $T_1$ to $T_2$ is the ratio between those efficacies $\mathrm{ARE}(T_1,T_2):=\frac{c_{T_1}^2}{c_{T_2}^2}$

Link to original

Signed-Rank Wilcoxon

Signed-Rank Wilcoxon Test

Definition

Let $X_1,X_2,\dots ,X_n$ be Random Sample from cdf $F(x)$, symmetric pdf $f(x)$, and median $\theta$. Consider a Sign Test $H_0:\theta =0,H_1:\theta >0$

The signed-rank Wilcoxon test statistic for the test is defined as $T=\sum _{i=1}^n\mathrm{sgn}(X_i)R|X_i|$ where $R|X_i|$ is the rank of $X_i$, among $|X_i|,\dots ,|X_n|$

We reject $H_0:\theta =0$ if $T\ge c$, where $c$ is determined by the level $\alpha$

$R|X_i|$ takes one of $1,2,\dots ,n$. So, we can rewrite the statistic $T$ as $T=\sum _{i=1}^{n}j\cdot \mathrm{sgn}(X_{ij})$ where $X_{ij}$ is an observation such that $R|X_i|=j$

Let $T^{+}$ be the sum of the ranks of positive $X_i$‘s, then $T=2T^{+}-\frac{n(n+1)}{2}$ So. $T^{+}=\frac{T}{2}+\frac{n(n+1)}{4}$ $E(T^{+})=\frac{n(n+1)}{4},\mathrm{Va}{\mathrm{r}}_{{\mathrm{H}}_0}(T^{+})=\frac{n(n+1)(2n+1)}{24}$

$T^{+}$ can be written as $T^{+}=\underset{i\le j}{\sum \sum }I\left(\frac{X_i+X_j}{2}>0\right)$ where $\frac{X_i+X_j}{2}$ are called the Walsh averages

In general, if the median of $X_i$ is $\theta$, then $T^{+}=\underset{i\le j}{\sum \sum }I\left(\frac{X_i+X_j}{2}>\theta \right)$

Consider a sequence of local alternatives $H_0:\theta =0,H_1:{\theta }_n=\delta /\sqrt{n},\delta >0$ The efficacy of the modified signed-rank Wilcoxon test statistic $T^{+}$ is $c_{T^{+}}={\tau }_{\omega }^{-1}=\sqrt{12}{\int }_{-\infty }^{\infty }{f}^2(x)dx$

Facts

Assume that $f(x)$ is symmetric, then under $H_0:\theta =0$, $|X_1|,|X_2|,\dots ,|X_n|$ are independent of $\mathrm{sgn}(X_1),\mathrm{sgn}(X_2),\dots ,\mathrm{sgn}(X_n)$

Let $X_1,X_2,\dots ,X_n$ be Random Sample from cdf $F(x)$, symmetric pdf $f(x)$, and median $\theta$, and $T=\sum _{i=1}^n\mathrm{sgn}(X_i)R|X_i|$, then under $H_0:\theta =0$

• $T$ is distribution free
• $E_{H_0}(T)=0$
• ${\mathrm{Var}}_{H_0}(T)=\frac{n(n+1)(2n+1)}{6}$
• $\frac{T}{\sqrt{{\mathrm{Var}}_{H_0}(T)}}\stackrel{D}{\to }N(0,1)$

Consider a sequence of hypothesis $H_0:\theta =0,H_1:{\theta }_n=\delta /\sqrt{n},\delta >0$, then ${\lim }_{n\to \infty }{\gamma }_{SR}({\theta }_n)=1-\Phi (z_a-\delta {\tau }_{\omega }^{-1})$ where $\Phi$ is a CDF of standard normal distribution, and ${\tau }_{\omega }=c_{T^{+}}^{-1}=\frac{1}{\sqrt{12}{\int }_{-\infty }^{\infty }{f}^2(x)dx}$

The estimator ${\hat{\theta }}_W$, which can be a solution of the equation $T^{+}({\hat{\theta }}_W)=\frac{n(n+1)}{4}$, is called Hodges-Lehmann estimator. Hodges-Lehmann estimator is an estimator of median of the Warsh average $\frac{X_i+X_j}{2}$.

Let $X_i=\theta +{ϵ}_i$ be random variables from location model, where ${ϵ}_i$‘s are i.i.d. with CDF $F(x)$, symmetric PDF $f(x)$, and median 0, then $\sqrt{n}({\hat{\theta }}_W-\theta )\stackrel{D}{\to }N(0,{\tau }_W^2)$ where ${\tau }_W=\frac{1}{\sqrt{12}{\int }_{-\infty }^{\infty }{f}^2(x)dx}$

Link to original

Mann-Whitney-Wilcoxon Procedure

Mann-Whitney-Wilcoxon Procedure

Definition

Let $X_1,X_2,\dots ,X_{n_1}$ be Random Sample from a distribution with cdf $F(x)$, and $Y_1,Y_2,\dots ,Y_{n_2}$ be Random Sample from a distribution with cdf $F(x-\Delta )$ Consider a test $H_0:\Delta =0,H_1:\Delta >0$

The Mann-Whitney-Wilcoxon test statistic for the test is defined as $W=\sum _{j=1}^{n_2}R(Y_j)$ where $R(Y_j)$ is the rank of $Y_j$ in the combined (pooled) sample of size $n=n_1+n_2$

We reject $H_0:\Delta =0$ when $W\ge c$, where $c$ is defined by the level $\alpha$

$R(Y_j)=\sum _{i=1}^{n_1}I(X_i<Y_i)+\sum _{k=1}^{n_2}I(Y_k\le Y_j)$ Therefore, the Mann-Whitney-Wilcoxon test statistic is decomposed as $W=\sum _{j=1}^{n_2}R(Y_j)=\sum _{i=1}^{n_1}\sum _{j=1}^{n_2}I(X_i<Y_j)+\sum _{k=1}^{n_2}\sum _{j=1}^{n_2}I(Y_k\le Y_j)=U+\frac{n_2(n_2+1)}{2}$ where $U:=\sum _{i=1}^{n_1}\sum _{j=1}^{n_2}I(X_i<Y_j)$

So, We reject $H_0:\Delta =0$ when $U\ge c$ $E(U)=\frac{n_1{n}_2}{2}$

Also, the power function of the test is defined as $U(\Delta )=\sum _{i=1}^{n_1}\sum _{j=1}^{n_2}I(X_i<Y_j-\Delta )=\sum _{i=1}^{n_1}\sum _{j=1}^{n_2}I(Y_j-X_i>\Delta )$

Consider a sequence of local alternatives $H_0:\Delta =0,H_1:{\Delta }_n=\delta /\sqrt{n},\delta >0$, and assume that $\frac{n_1}{n}\to {\lambda }_1,\frac{n_2}{n}\to {\lambda }_2$, where ${\lambda }_1+{\lambda }_2=1$ The efficacy of the modified Mann-Whitney-Wilcoxon test statistic $U$ is $c_U={\tau }_W^{-1}=\sqrt{12}\sqrt{{\lambda }_1{\lambda }_2}{\int }_{-\infty }^{\infty }{f}^2(x)dx$

Facts

Let $X_1,X_2,\dots ,X_{n_1}$ be Random Sample from a distribution with cdf $F(x)$, $Y_1,Y_2,\dots ,Y_{n_2}$ be Random Sample from a distribution with cdf $F(x-\Delta )$, and $W=\sum _{j=1}^{n_2}R(Y_j)$ then under $H_0:\Delta =0$

• $W$ is distribution free
• $E_{H_0}(W)=\frac{n_2(n+1)}{2}$
• ${\mathrm{Var}}_{H_0}(W)=\frac{n_1{n}_2(n+1)}{12}$
• $\frac{W-n_2(n+1)/2}{\sqrt{{\mathrm{Var}}_{H_0}(W)}}\stackrel{D}{\to }N(0,1)$

Consider a sequence of hypothesis $H_0:\Delta =0,H_1:{\Delta }_n=\delta /\sqrt{n},\delta >0$, then ${\lim }_{n\to \infty }{\gamma }_U({\Delta }_n)=1-\Phi (z_a-\sqrt{{\lambda }_1{\lambda }_2}\delta {\tau }_W^{-1})$ where $\Phi$ is a CDF of standard normal distribution, and ${\tau }_W=c_U^{-1}=\frac{1}{\sqrt{12}\sqrt{{\lambda }_1{\lambda }_2}{\int }_{-\infty }^{\infty }{f}^2(x)dx}$

Link to original

Measures of Association

Kendall's Tau

Definition

Let $(X_1,X_1),(X_2,X_2),\dots ,(X_n,X_n)$ be a Random Sample from a bivariate distribution with cdf $F(x,y)$ We want to test $H_0:X,Y$ are independent

Let $(X_1,X_1),(X_2,X_2)$ be independent pairs with the same bivariate distribution. If $\mathrm{sgn}\{(X_1-X_2)(Y_1-Y_2)\}=1$, then we say these pairs are concordant If $\mathrm{sgn}\{(X_1-X_2)(Y_1-Y_2)\}=-1$, then pairs are discordant

Kendall’s $\tau$ is defined as $\tau =P[\mathrm{sgn}\{(X_1-X_2)(Y_1-Y_2)\}=1]-P[\mathrm{sgn}\{(X_1-X_2)(Y_1-Y_2)\}=-1]$

$\tau$ is bounded by $\pm 1$. If $X,Y$ are independent, then $\tau =0$

Consider another statistic $K={\left(\genfrac{}{}{0ex}{}{n}{2}\right)}^{-1}\underset{i<j}{\sum \sum }\mathrm{sgn}\{(X_1-X_2)(Y_1-Y_2)\}$

$E[K]=\tau$, so it is an unbiased estimator of $\tau$

Facts

Let $(X_1,X_1),(X_2,X_2),\dots ,(X_n,X_n)$ be a Random Sample from a bivariate distribution with cdf $F(x,y)$, and $K={\left(\genfrac{}{}{0ex}{}{n}{2}\right)}^{-1}\underset{i<j}{\sum \sum }\mathrm{sgn}\{(X_1-X_2)(Y_1-Y_2)\}$, then under $H_0:X,Y$ are independent

• $K$ is distribution free with a symmetric pdf
• $E_{H_0}(K)=0$
• ${\mathrm{Var}}_{H_0}(K)=\frac{2(2n+1)}{9n(n-1)}$
• $\frac{K}{\sqrt{{\mathrm{Var}}_{H_0}(K)}}\stackrel{D}{\to }N(0,1)$

Link to original

Spearman's Rho

Definition

Let $(X_1,X_1),(X_2,X_2),\dots ,(X_n,X_n)$ be a Random Sample from a bivariate distribution with cdf $F(x,y)$

Instead of a sample Pearson Correlation Coefficient, we define another statistic using the ranks of samples $r_S=\frac{{\sum }_{i=1}^n(R(X_i)-\frac{n+1}{2})(R(Y_i)-\frac{n+1}{2})}{n(n^2-1)/12}$ which is called Spearman’s $\rho$

$r_s$ is bounded by $\pm 1$. If $X,Y$ are independent, then $r_S=0$

Facts

Let $(X_1,X_1),(X_2,X_2),\dots ,(X_n,X_n)$ be a Random Sample from a bivariate distribution with cdf $F(x,y)$, and $r_S=\frac{{\sum }_{i=1}^n(R(X_i)-\frac{n+1}{2})(R(Y_i)-\frac{n+1}{2})}{n(n^2-1)/12}$, then under $H_0:X,Y$ are independent

• $r_S$ is distribution free with a symmetric pdf
• $E_{H_0}(r_S)=0$
• ${\mathrm{Var}}_{H_0}(r_S)=\frac{1}{n-1}$
• $\frac{r_S}{\sqrt{{\mathrm{Var}}_{H_0}(r_S)}}\stackrel{D}{\to }N(0,1)$

Link to original