Probability Theory Note

Random Variable

Definition

$X:\Omega \to \mathbb{R}\text{s.t.}\forall B\in \mathcal{R},\exists \omega \in \Omega ,\text{s.t.}{X}^{-1}(B)=\{\omega :X(\omega )\in B\}\in \mathcal{F}$

A random variable is a function $X:\Omega \to \mathbb{R}$ whose inverse function $X^{-1}(B)$ is $\mathcal{F}$-measurable for the two measurable spaces $(\Omega ,\mathcal{F})$ and $(\mathbb{R},\mathcal{R})$.

The inverse image of an arbitrary Borel set of Codomain $\mathbb{R}$ is an element of sigma field $\mathcal{F}$.

Notations

Consider a probability space $(\Omega ,\mathcal{F},\mathbb{P})$

• Outcomes: $H,T$
• Set of outcomes (Sample space): $\Omega =\{H,T\}$
• Events: $\varnothing ,\{H\},\{T\},\Omega$
• Set of events (Sigma-Field): $\mathcal{F}=\{\varnothing ,\{H\},\{T\},\{H,T\}\}$
• Probabilities: $\mathbb{P}:\mathcal{F}\to [0,1]$
• Random variable: $X:\Omega \to \mathbb{R}$

For a random variable $X$ on a Probability Space $(\Omega ,\mathcal{F},P)$

• $X\sim {\mu }_X$ if and only if the Distribution of $X$ is ${\mu }_X$
• $X\sim F_X$ if and only if the Distribution Function of $X$ is $F_X$

For a random variable $X$ on Probability Space $({\Omega }_X,{\mathcal{F}}_X,P_X)$ and another random variable $Y$ on Probability Space $({\Omega }_Y,{\mathcal{F}}_Y,P_Y)$

• $X\stackrel{d}{=}Y⟺\forall B\in \mathcal{R}:{\mu }_X(B)={\mu }_Y(B)$
• $X\stackrel{d}{=}Y⟺\forall k\in \mathbb{R}:F_X(k)=F_Y(k)$
• $X\stackrel{d}{=}Y⟺\forall k\in \mathbb{R}:P_X(X\le k)=P_Y(Y\le k)$

Link to original

Distribution

Definition

${\mu }_X:=P\circ X^{-1}:\mathcal{R}\to [0,1]$ where $\mathcal{R}$ is a Borel Sigma-Field

A distribution ${\mu }_X$ is a function for the Random Variable $X$ on Probability Space $(\Omega ,\mathcal{F},P)$

Facts

${\mu }_X$ is a Probability Measure on $(\mathbb{R},\mathcal{R})$, so $(\mathbb{R},\mathcal{R},{\mu }_X)$ is a Probability Space. The Probability Space $(\mathbb{R},\mathcal{R},{\mu }_X)$ is called the Probability Space induced by $X$

Link to original

Distribution Function

Definition

$F_X(x):\mathbb{R}\to [0,1]:={\mu }_X((-\infty ,x])=P(X\le x)$

A distribution function $F_X$ is a function for the Random Variable on Probability Space $(\Omega ,\mathcal{F},P)$

Facts

$(\Omega ,\mathcal{F},P)\sim (\mathbb{R},\mathcal{R},{\mu }_X)$

Proof By definition, $F_X(x)={\mu }_X((-\infty ,x])$ So, defining $\forall x\in \mathbb{R},F_X(x)$ is the Equivalence Relation to defining $\mathcal{A}=\{(-\infty ,x]:x\in \mathbb{R}\},{\mu }_X:\mathcal{A}\to [0,1]$

Since $\mathcal{A}$ is a Pi-System, ${\mu }_X:\sigma (\mathcal{R})=\mathcal{R}\to [0,1]$ is uniquely determined by ${\mu }_X:\mathcal{A}\to [0,1]$ by Extension from Pi-System

Now, $(\mathbb{R},\mathcal{R},{\mu }_X)$ is a Measurable Space induced by $X$. Therefore, defining ${\mu }_X$ on $(\mathbb{R},\mathcal{R})$ is equivalent to the defining $P$ on $(\Omega ,\mathcal{F})$

Distribution function(CDF) $F:\mathbb{R}\to [0,1]$ has the following properties

• Monotonic increasing: $\forall a,b,\in \mathbb{R},a<b\Rightarrow F(a)\le F(b)$
• $\underset{x\to -\infty }{\lim }F(x)=0$
• $\underset{x\to \infty }{\lim }F(x)=1$
• Right-continuous: $\underset{x\to x_0^{+}}{\lim }F(x)=F(x_0)$

If a function $F:\mathbb{R}\to [0,1]$ satisfies the following properties, then $F$ is a distribution function(CDF) of some Random Variable $X$

• Monotonic increasing: $\forall a,b,\in \mathbb{R},a<b\Rightarrow F(a)\le F(b)$
• $\underset{x\to -\infty }{\lim }F(x)=0$
• $\underset{x\to \infty }{\lim }F(x)=1$
• Right-continuous: $\underset{x\to x_0^{+}}{\lim }F(x)=F(x_0)$

$\forall a,b\in \mathbb{R},a<b\Rightarrow (P(a<X\le b)=F_X(b)-F_X(a))$

Link to original

Transclude of Density-Function

Extended Real Number Space

Definition

$\bar{\mathbb{R}}=\mathbb{R}\cup \{-\infty \}\cup \{\infty \}$

Link to original

Measurable Function

Definition

$f:\Omega \to S\text{s.t.}\forall B\in \mathcal{S},f^{-1}(B)=\{\omega \in \Omega |f(\omega )\in B\}\in \mathcal{F}$

A measurable function is a function $f:\Omega \to S$ whose inverse function $X^{-1}(B)$ is $\mathcal{F}$-measurable the two measurable spaces $(\Omega ,\mathcal{F}),(S,\mathcal{S})$.

Other Expressions

$f:\Omega \to S$ is a measurable function on the two measurable spaces $(\Omega ,\mathcal{F}),(S,\mathcal{S})$ $f:(\Omega ,\mathcal{F})\to (S,\mathcal{S})$ $f$ is $\mathcal{F}-\mathcal{S}$ measurable $f\in \mathcal{F}$ (only if $(S,\mathcal{S})=(\mathbb{R},\mathcal{R})$) $f$ is a $S$-valued measurable

Examples

When $(S,\mathcal{S})=(\mathbb{R},\mathcal{R})$, the measurable function is called a Random Variable. When $(S,\mathcal{S})=({\mathbb{R}}^d,{\mathcal{R}}^d)$, the measurable function is called a Random Vector.

Facts

For the $\mathcal{F}$-measurable set $A\in \mathcal{F}$ on the two measurable spaces $(\Omega ,\mathcal{F}),(\mathbb{R},\mathcal{R})$ A function $f:\Omega \to \mathbb{R}$ defined as $f(\omega )=\{\begin{array}{ll}1& \omega \in A\\ 0& \omega \notin A\end{array}$ is a $\mathcal{F}-\mathcal{R}$ measurable function

If $f$ is a $\mathbb{R}$-valued measurable function on the two measurable spaces $(\Omega ,\mathcal{F}),(\mathbb{R},\mathcal{R})$, then $\forall \alpha \in \mathbb{R},\alpha f(\omega )$ also $\mathbb{R}$-valued measurable. Scaling of function does not change the Domain, only scale the Image

If $f:(\Omega ,\mathcal{F})\to (S,\mathcal{S})$ and $g:(S,\mathcal{S})\to (T,\mathcal{T})$ on the three measurable spaces $(\Omega ,\mathcal{F}),(S,\mathcal{S}),(T,\mathcal{T})$, then $g\circ f:(\Omega ,\mathcal{F})\to (T,\mathcal{T})$

If a function $f:\mathbb{R}\to \mathbb{R}$ is continuous, is measurable.

If two functions $f,g$ are $\mathbb{R}$-valued measurable on the two measurable spaces $(\Omega ,\mathcal{F}),(\mathbb{R},\mathcal{R})$, then following are all $\mathbb{R}$-valued measurable

• $h:=f+g$
• $h:=f-g$
• $h:=\max (f,g)$, where $h:\omega \mapsto \max (f(\omega ),g(\omega ))$
• $h:=\min (f,g)$, where $h:\omega \mapsto \min (f(\omega ),g(\omega ))$

If every element of a function sequence $\{f_n:n\in \mathbb{N}\}$ is $\mathbb{R}$-valued measurable, then the following are all $\mathbb{R}$-valued measurable.

• $h:=\inf f_n$, where $h:\omega \mapsto \inf \{f_1(\omega ),f_2(\omega ),\dots \}$
• $h:=\sup f_n$, where $h:\omega \mapsto \sup \{f_1(\omega ),f_2(\omega ),\dots \}$

Link to original

Lebesgue Integral Note

Almost Everywhere

Definition

The Propositional Function is satisfied for every point except where Lebesgue Measure is 0

Link to original

Lebesgue Integral

Definition

Riemann integral and Lebesgue integral

Consider a measure spaces $(\Omega ,\mathcal{F},\mu )$ and a Measure Space $(\overline{\mathbb{R}},\mathcal{R})$, where $\mathcal{R}$ is the Borel Sigma-Field on $\overline{\mathbb{R}}$, and a Measurable Function $f:(\Omega ,\mathcal{F})\to (\overline{\mathbb{R}},\mathcal{R})$. Then the Lebesgue integral of the function $f$ with respect to $\mu$ is denoted as:

${\int }_{\Omega }fd\mu ={\int }_{\Omega }f(x)d\mu (x)={\int }_{\Omega }f(x)\mu (dx)$

Approximation of Lebesgue Integral with Finite Sum

$\int fd\lambda \approx \sum _{i=1}^N{a}_n\lambda (A_n)$ where $A_n:=\{x|f(x)=a_n\}$ or $A_n=f^{-1}(\{a_n\})$.

To compute the Riemann Integral of $f$, one partitions the domain $[a,b]$ into sub-intervals, while in the Lebesgue integral, one partitions the image of $f$.

Link to original

Simple Function

Definition

$f(\omega )=\sum _{i=1}^n{\alpha }_i{1}_{A_i}(\omega )$ where $A_i\in \mathcal{F}$ are disjoint measurable sets, $a_i$ are the sequence of real or complex numbers, and $1_A$ is a Indicator Function

Finite linear combination of indicator functions of measurable sets

Facts

Simple function $f$ is $\mathcal{F}-\mathcal{R}$ measurable map

A Measurable Function $f$ with a finite Image is a Simple Function

Link to original

Lebesgue Integral of Simple Function

Definition

Consider a measure spaces $(\Omega ,\mathcal{F},\mu )$ and a Measure Space $(\overline{\mathbb{R}},\mathcal{R})$, and a positive $\mathcal{F}-\mathcal{R}$ measurable Simple Function, $f:(\Omega ,\mathcal{F})\to (\overline{\mathbb{R}},\mathcal{R})$, with Codomain $R:=\{r_1,r_2,\dots ,r_n\}$ that have corresponding preimages $f^{-1}(r_i)=A_i\in \mathcal{F}$. Then the Lebesgue integral of the function $f$ with respect to $\mu$ is defined as ${\int }_{\Omega }fd\mu ={\sum }_{i=1}^n{r}_i\mu (A_i)$

Facts

Consider a measure spaces $(\Omega ,\mathcal{F},\mu )$ and a Measure Space $(\overline{\mathbb{R}},\mathcal{R})$, and a positive $\mathcal{F}-\mathcal{R}$ measurable Simple Function, $f:(\Omega ,\mathcal{F})\to (\overline{\mathbb{R}},\mathcal{R})$, with Codomain $R:=\{r_1,r_2,\dots ,r_n\}$ that have corresponding preimages $f^{-1}(r_i)=A_i\in \mathcal{F}$. Then, $f={\sum }_{i=1}^m{\alpha }_i{1}_{A_i}={\sum }_{j=1}^n{\beta }_j{1}_{B_j}⟹\int fd\mu ={\sum }_{i=1}^n{\alpha }_i\mu (A_i)={\sum }_{i=1}^n{\beta }_i\mu (B_i)$

Link to original

Lebesgue Integral of Non-Negative Function

Definition

Consider a measure spaces $(\Omega ,\mathcal{F},\mu )$ and a Measure Space $(\overline{\mathbb{R}},\mathcal{R})$ and a positive Measurable Function $f:(\Omega ,\mathcal{F})\to (\overline{\mathbb{R}},\mathcal{R})$. Then the Lebesgue integral of the function $f$ with respect to $\mu$ is defined as ${\int }_{\Omega }fd\mu :=\sup \left\{{\int }_{\Omega }\phi d\mu \right|0\le \phi \le f,\text{a.e. w.r.t.}\mu \}$ where each $\phi :(\Omega ,\mathcal{F})\to (\mathbb{R},\mathcal{R})$ is a Simple Function.

Facts

$\int fd\mu$ can be infinity

Link to original

Lebesgue Integral of Measurable Function

Definition

Consider a measure spaces $(\Omega ,\mathcal{F},\mu )$ and a Measure Space $(\overline{\mathbb{R}},\mathcal{R})$ and a Measurable Function $f:(\Omega ,\mathcal{F})\to (\overline{\mathbb{R}},\mathcal{R})$. Then the Lebesgue integral of the function $f$ with respect to $\mu$ is defined as ${\int }_{\Omega }fd\mu :={\int }_{\Omega }{f}^{+}d\mu -{\int }_{\Omega }{f}^{-}d\mu$ where $f^{+}:=\max (0,f)$ and $f^{-}:=\max (0,-f)$ for the function $f$.

Existence of Integral

$\int fd\mu$ exists

• $\int f^{+}d\mu <\infty$ and $\int f^{-}d\mu <\infty$ ($f$ is integrable w.r.t $\mu$) $\int fd\mu =\int f^{+}d\mu -\int f^{-}d\mu$
• $\int f^{+}d\mu =\infty$ and $\int f^{-}d\mu <\infty$ $\int fd\mu =\infty$
• $\int f^{+}d\mu <\infty$ and $\int f^{-}d\mu =\infty$ $\int fd\mu =-\infty$

$\int fd\mu$ not exists

• $\int f^{+}d\mu =\infty$ and $\int f^{-}d\mu =\infty$ $\int fd\mu$ is not defined

Notations

For a Density Function $F_X={\mu }_X((-\infty ,x])$ on the Measurable Space $(\mathbb{R},\mathcal{R},{\mu }_X)$ $\int gd{\mu }_X=\int gd{F}_X=\int g(x)d{F}_X(x)$

For a Density Function $F_X={\mu }_X((-\infty ,x])$ that has Density Function $f_X(x)$ on the Measurable Space $(\mathbb{R},\mathcal{R},{\mu }_X)$ $\int g(x)d{F}_X(x)=\int g(x)f_X(x)dx$

Consider a Measure Space $(\Omega ,\mathcal{F},\#)$ where $\Omega$ is a countable set, $\mathcal{F}=2^{\Omega }$ is a Sigma-Field defined on the set $\Omega$, and $\#$ is a Counting measure on $\mathcal{F}$. Then $\int fd\#=\sum _{i\in \Omega }f(i)$

Facts

For the functions $f^{+}:=\max (0,f)$ and $f^{-}:=\max (0,-f)$ of $f$,

• $f^{+},f^{-}$ are non-negative
• $|f|=f^{+}+f^{-}$
• $f=f^{+}-f^{-}$
• If $f$ is measurable, then $f^{+}$ and $f^{-}$ are also measurable.

Link to original

Link to original

Cantor Set

Definition

$\mathcal{C}:=\underset{n\to \infty }{\lim }{C}_n={\bigcap }_{n=0}^{\infty }{C}_n$ where $C_0:=[0,1]$ and $C_n:=\frac{C_{n-1}}{3}\cup \left(\frac{2}{3}+\frac{C_{n-1}}{3}\right)$

Cantor set is created by iteratively deleting the open middle third from a set of line segments.

Facts

The elements of $C$ are numbers consisting of 0 or 2 in ternary

$|\mathcal{C}|=|[0,1]|=2^{\aleph }$ The Cardinality of $\mathcal{C}$ is equal to the interval $[0,1]$

$\lambda (\mathcal{C})=0$ $\mathcal{C}$ is a Lesbesgue-measurable set, and its Lebesgue Measure is $0$

Link to original

Cantor Function

Definition

$F(x)=\underset{n\to \infty }{\lim }{F}_n(x)$

A function $F$ that has positive derivative on Cantor Set $\mathcal{C}$ and zero derivative on $[0,1]-\mathcal{C}$

Facts

Consider the Cantor function $F$

• $F$ is a Constant Function on $[0,1]-\mathcal{C}$
• $F$ has zero derivative Almost Everywhere
• $F$ is a non-negative function

Consider the Cantor function $F$

• $F$ is monotonically increasing
• $F(0)=0$
• $F(1)=1$
• $F$ is continuous everywhere

Thus by the property of distribution function, there exists a continuous Random Variable corresponding to $F$.

$F$ has a derivative at Almost Everywhere, but the integral of the derivative and $F$ is not the same.

Link to original

Absolute Continuity

Definition

$\mu \ll \lambda$ Consider measures $\mu ,\lambda$ on a Measurable Space $(\Omega ,\mathcal{F})$. The measure $\mu$ said to be absolutely continuous with respect to $\lambda$, or $\mu$ is dominated by $\lambda$, denoted by $\mu \ll \lambda$, If $\forall S\in \mathcal{F},\lambda (S)=0\Rightarrow \mu (S)=0$

Summary

Comparison of conditions

Function	Derivative	Radon–Nikodym Derivative
Continuous Function	X	X (Cantor Function)
Absolute continuous function	X	O
Differentiable function	O	O

• Differentiable function $\subset$ Absolute continuous function $\subset$ Continuous function

Facts

Continuously differentiable $\subset$ Lipschitz continuous $\subset$ $\alpha -$Holder Continuous $\subset$ Uniformly Continuous $\subset$ Continuous Lipschitz continuous Absolute continuous $\subset$ Uniformly Continuous $\subset$ Continuous where $0<0\le 1$

Link to original

Link to original

Radon–Nikodym Theorem

Definition

Consider sigma-finite measures $\mu ,\lambda$ on a Measurable Space $(S,\mathcal{S})$. If $\mu \ll \lambda$ ($\mu$ is absolutely continuous with respect to $\lambda$), then there exists a $\mathcal{S}$-Measurable Function $f:X\to {\mathbb{R}}_0^{+}$ such that $\forall B\in \mathcal{S},\mu (B)={\int }_{B}fd\lambda$ almost uniquely with respect to $\lambda$. Where the function $f:(S,\mathcal{R})\to ({\mathbb{R}}^{+},{\mathcal{R}}^{+})$ is called a Radon-Nikodym derivative of $\mu$ w.r.t. $\lambda$ $\mu \ll \lambda \Rightarrow \exists !f=\frac{d\mu }{d\lambda }\text{s.t.}\forall B\in \mathcal{S},\mu (B)={\int }_{B}fd\lambda$

Radon–Nikodym Derivative

$f=\frac{d\mu }{d\lambda }$ Consider sigma-finite measures $\mu ,\lambda$ on a Measurable Space $(S,\mathcal{S})$. A Radon–Nikodym derivative $f=\frac{d\mu }{d\lambda }$ is a function satisfying $\forall B\in \mathcal{S},\mu (B)={\int }_{B}fd\lambda$.

Creating New Measure

Given a measure space $(S,\mathcal{S},\lambda )$ and a non-negative Measurable Function $f:S\to {\mathbb{R}}_0^{+}$, we can define a new measure $\mu$ by the relation $\forall B\in S,\mu (B)={\int }_{B}fd\lambda$ The measure $\mu$ is absolutely continuous with respect to $\lambda$ ($\mu \ll \lambda$) and $f$ be a Radon-Nikodym derivative of $\mu$ with respect to $\lambda$.

Examples

Distribution Case

Consider a distribution $F_X:\mathbb{R}\to [0,1]$ and a Lebesgue Measure $\lambda$, and a Measurable Space $(\mathbb{R},\mathcal{R})$ (Borel Sigma-Field on Real Number). If ${\mu }_X\ll \lambda$, then there exists a Density Function $f_X$ of the Distribution Function $F_X$ ${\mu }_X\ll \lambda \Rightarrow \exists !f_X=\frac{d{\mu }_X}{d\lambda }\text{s.t.}{F}_X={\int }_{(-\infty ,x]}{f}_{X}d\lambda$

Facts

If another function $g$ have the same properties, then $f=g$ at Almost Everywhere

Link to original

Generalized Density Function

category	measurable space	distribution	measure	description
Continuous	$(\mathbb{R},\mathcal{R})$	${\mu }_X$	$\lambda$
Discrete	$(S,\mathcal{S})$	${\tilde{\mu }}_X:\mathcal{S}\to [0,1]$	$\#$: Counting measure	$\#$ is not sigma-finite on $(\mathbb{R},\mathcal{R})$
Discrete	$(\mathbb{R},\mathcal{R})$	${\mu }_X$	${\#}_X$	${\#}_X$ is sigma-finite on $(\mathbb{R},\mathcal{R})$

Where:

• $S$ is a support of a discrete random variable
• $\mathcal{S}=2^S$.
• ${\#}_X:={\sum }_{i\in X}{\delta }_i$: only counts the number of elements in the support of a discrete random variable $X$. It can be expressed as the sum of Dirac measure

Link to original

Lebesgue Decomposition Theorem

Definition

A Distribution $F$ can be decomposed as the following $F=F_{ac}+F_{pp}+F_{sing}$ Where:

• $F_{ac}$: absolutely continuous part (absolutely continuous w.r.t $\lambda$)
• $F_{pp}$: pure point part: (absolutely continuous w.r.t ${\#}_X$)
• $F_{sing}$: singular continuous part (e.g. cantor distribution)

Facts

$F_{pp}$ has countable discontinuous points

If a Distribution Function $F=F_{ac}$, then there exists corresponding continuous Random Variable $X$ and PDF

If a Distribution Function $F=F_{pp}$, then there exists corresponding discrete Random Variable $X$ and generalized PDF or PMF

If a Distribution Function $F=F_{ac}+F_{pp}$, then there exists corresponding mixed Random Variable $X$ and generalized PDF

Link to original

Expected Value

Definition

$E(X)={\int }_{\Omega }XdP={\int }_{\mathbb{R}}xd{\mu }_X={\int }_{\mathbb{R}}xd{F}_X$ The expected value of the Random Variable $X$ on Probability Space $(\Omega ,\mathcal{F},P)$

Continuous

$E(X)={\int }_{-\infty }^{\infty }x{f}_X(x)dx$ where $f_X(x)$ is a PDF of Random Variable $X$

The expected value of the Random Variable $X$ when $F_X$ satisfies absolute continuous over $\lambda$, ${\mu }_X<<\lambda$

Discrete

$E(X)={\sum }_{x}x{p}_X(x)$ where $p_X(x)$ is a PMF of Random Variable $X$

The expected value of the Random Variable $X$ when $F_X$ satisfies absolute continuous over $\#$, ${\mu }_X<<{\#}_X$ In other words, $F_X$ has at most countable jumps.

Expected Value of a Function

$E(g(x))={\int }_{\mathbb{R}}g(x)d{\mu }_X$

Continuous

$E(g(x))={\int }_{\mathbb{R}}g(x)f_X(x)dx$

Discrete

$E(g(x))=\sum _{x\in S_X}g(x)p_X(x)$

Properties

Linearity

Random Variables

If $\exists k_{i}E[X_i]$, then $E\left[{\sum }_{i=1}^n{k}_i{X}_i\right]={\sum }_{i=1}^n{k}_{i}E(X_i)$

Matrix of Random Variables

Let $W_1,W_2$ be a $m\times n$ matrices of random variables, $A_1,A_2$ be $k\times m$ matrices of constants, and $B$ a $n\times l$ matrix of constant. Then, $E[A_1{W}_1+A_2{W}_2]=A_{1}E[W_1]+A_{2}E[W_2]$

$E[A_1{W}_{1}B]=A_{1}E[W_1]B$

Notations

Expression	Discrete	Continuous
Expression for the event $2^{\Omega }$ and the probability $P$	${\sum }_{\omega \in \Omega }X(\omega )\cdot P(\omega )$	${\int }_{\Omega }X(\omega )\cdot dP(\omega )$
Expression for the measurable space $(\mathbb{R},\mathcal{R})$ and the distribution ${\mu }_X$	${\sum }_{x\in \mathbb{R}}x\cdot f(x)$	${\int }_{\mathbb{R}}f(x)\lambda (x)={\int }_{\mathbb{R}}\frac{d{\mu }_X}{d\lambda }\lambda (x)={\int }_{\mathbb{R}}d{\mu }_X={\int }_{\mathbb{R}}d{F}_X$

Link to original