Advanced Probability Theory Note

Measure Theory Note

Algebraic Structure (Measure Theory)

Kinds

Sigma-Algebra

Definition

A sigma-algebra $\mathcal{F}$ is a Family of Sets satisfying the following properties

• $\Omega \in \mathcal{F}\text{or}\mathcal{F}\ne \varnothing$
• $\forall A\in \mathcal{F},A^{\complement }\in \mathcal{F}$
• $\forall \{A_i{\}}_{i=1}^{\infty }\subset \mathcal{F},\bigcup _{i=1}^{\infty }{A}_i\in \mathcal{F}$

where $\Omega$ is a universal set

Intersection of Sigma-Fields

Consider a set of sigma-fields ${\mathcal{F}}_i$ on $\Omega$,

• ${\mathcal{F}}_1\cap {\mathcal{F}}_2$ is a sigma-field on $\Omega$.
• $\bigcup _{i=1}^{\infty }{\mathcal{F}}_i$ is a sigma-field on $\Omega$.
• $\bigcup ^{\infty }{\mathcal{F}}_i$ is a sigma-field on $\Omega$

Facts

If a set of subsets $\mathcal{A}$ of a universal set $\Omega$ is both Pi-System and Lambda-System, then $\mathcal{A}$ is a sigma-algebra on $\Omega$.

Link to original

Algebra of Sets

Definition

An algebra $\mathcal{A}$ is a Family of Sets satisfying the following properties

• $\Omega \in \mathcal{A}$
• $\forall A,B\in \mathcal{A},A-B\in \mathcal{A}$
• $\forall A,B\in \mathcal{A},A\cup B\in \mathcal{A}$

where $\Omega$ is a universal set

The conditions imply:

• $\varnothing \in \mathcal{R}$.

Equivalents

• $\Omega \in \mathcal{A}$
• $\forall A\in \mathcal{A},A^{\complement }\in \mathcal{A}$
• $\forall A,B\in \mathcal{F},A\cap B\in \mathcal{A}$

• $\Omega \in \mathcal{A}$
• $\forall A\in \mathcal{A},A^{\complement }\in \mathcal{A}$
• $\forall A,B\in \mathcal{F},A\cup B\in \mathcal{A}$

Link to original

Ring of Sets (Measure Theory)

Definition

A ring $\mathcal{R}$ is a Family of Sets satisfying the following properties

• $\forall A,B\in \mathcal{R},A-B\in \mathcal{R}$
• $\forall A,B\in \mathcal{R},A\cup B\in \mathcal{R}$

This implies the following

• $\varnothing \in \mathcal{R}$
• $\forall A,B\in \mathcal{R},A\cap B\in \mathcal{R}\because A\cap B=A-(A-B)$

Link to original

Sigma-Ring

Definition

A sigma-ring $\mathcal{R}$ is a Family of Sets satisfying the following properties

• $\forall A,B\in \mathcal{R},A-B\in \mathcal{R}$
• $\forall \{A_i{\}}_{i=1}^{\infty }\subset \mathcal{R},\bigcup _{i=1}^{\infty }{A}_i\in \mathcal{R}$

Link to original

Semi-Algebra

Definition

A semi-algebra $\mathcal{A}$ is a Family of Sets satisfying the following properties

• $\Omega \in \mathcal{A}$
• $\forall A,B\in \mathcal{A},A\cap B\in \mathcal{A}$
• $\forall A,B\in \mathcal{A},\exists \{B_i{\}}_{i=1}^n\subset \mathcal{A}\text{s.t.}A-B=\underset{i=1}{\overset{n}{⨄}}{B}_i\text{or}\forall A\in \mathcal{A},\exists \{B_i{\}}_{i=1}^n\subset \mathcal{A}\text{s.t.}{A}^{\complement }=\underset{i=1}{\overset{n}{⨄}}{B}_i$

where $\Omega$ is a universal set

Link to original

Semiring

Definition

A semiring $\mathcal{S}$ is a Family of Sets satisfying the following properties

• $\varnothing \in \mathcal{S}$
• $\forall A,B\in \mathcal{S},A\cap B\in \mathcal{S}$
• $\forall A,B\in \mathcal{S},\exists \{B_i{\}}_{i=1}^n\subset \mathcal{S}\text{s.t.}A-B=\underset{i=1}{\overset{n}{⨄}}{B}_i$

where $\uplus$ is a union of the pairwise disjoint sets

Examples

The all below are the semiring on $\Omega =\mathbb{R}$

• $\mathcal{A}=\{(a,b]:-\infty <a<b<\infty \}\cup \{\varnothing \}$
• $\mathcal{A}=\{[a,b):-\infty <a<b<\infty \}\cup \{\varnothing \}$

Link to original

Pi-System

A $\pi$-system $\mathcal{P}$ is a Family of Sets satisfying the following property

• $\forall A,B\in \mathcal{P},A\cap B\in \mathcal{P}$

Examples

The all below are the pi-system on $\Omega =\mathbb{R}$

• $\mathcal{A}=\{(a,b]:-\infty <a<b<\infty \}\cup \{\varnothing \}$
• $\mathcal{A}=\{[a,b):-\infty <a<b<\infty \}\cup \{\varnothing \}$
• $\mathcal{A}=\{(a,b):-\infty <a<b<\infty \}\cup \{\varnothing \}$
• $\mathcal{A}=\{[a,b]:-\infty <a<b<\infty \}\cup \{\varnothing \}$

Link to original

Transclude of Lambda-System

Summary

	$A\cap B$	$\varnothing$	$A-B$	$\bigcup _i{A}_i\to \underset{i}{⨄}{B}_i$	$\Omega$	$A^c$	$A\cup B$	$\bigcup _{i=1}^{\infty }{A}_i$	$\underset{i=1}{\overset{\infty }{⨄}}{B}_i$	$\bigcap _{i=1}^{\infty }{A}_i$
$\pi$-system	$O$	$X$	$X$	$X$	$X$	$X$	$X$	$X$	$X$	$X$
semi-ring	$O$	$O$	$\Delta$	$O$	$X$	$X$	$X$	$X$	$X$	$X$
semi-algebra	$O$	$O$	$\Delta$	$O$	$O$	$\Delta$	$X$	$X$	$X$	$X$
ring	$O$	$O$	$O$	$O$	$X$	$X$	$O$	$X$	$X$	$X$
algebra	$O$	$O$	$O$	$O$	$O$	$O$	$O$	$X$	$X$	$X$
$\sigma$-ring	$O$	$O$	$O$	$O$	$X$	$X$	$O$	$O$	$O$	$O$
$\lambda$-system	$X$	$O$	$\Delta$’	$X$	$O$	$O$	$X$	$X$	$O$	$X$
$\sigma$-field	$O$	$O$	$O$	$O$	$O$	$O$	$O$	$O$	$O$	$O$

${\Delta }^{\prime }$: $A\subset B\Rightarrow B-A\in \mathcal{D}$

Diagrams

flowchart TD
 
subgraph ALGEBRA
sia["$\sigma$-algebra"]
a[algebra]
sea[semialgebra]
 
sia --> a
a --> sea
end
 
subgraph RING
sir["$\sigma$-ring"]
r[ring]
ser[semiring]
 
sir --> r
r --> ser
end
 
subgraph LAMBDA
ls["$\lambda$-system"]
end
 
ps["$\pi$-system"]
 
sia --> sir
sia --> ls
 
a --> r
 
sea --> ser
 
ser --> ps

flowchart TD
 
ps["$\pi$-system"]
 
subgraph RING
ser[semiring]
r[ring]
sir["$\sigma$-ring"]
 
ser -->|"$\cup$-stable"|r
r -->|"$\sigma$-$\cup$-stable"|sir
end
 
subgraph ALGEBRA
sea[semialgebra]
a[algebra]
sia["$\sigma$-algebra"]
 
sea -->|"$\cup$-stable"|a
a -->|"$\sigma$-$\cup$-stable"|sia
end
 
subgraph LAMBDA
ls["$\lambda$-system"]
 
ls -->|"$\cap$-stable"|sia
end
 
ps -->|"$\uplus$-stable"|ser
 
ser -->|"$\Omega$-contained"|sea
r -->|"$\Omega$-contained"|a
sir -->|"$\Omega$-contained"|sia
 

Link to original

Measure

Definition

A function defined on a measurable space $(\Omega ,\mathcal{F})$, in which satisfies the following conditions

• Non-negativity: $\mu :\mathcal{F}\to {\mathbf{R}}_0^{+}$
• $\mu (\varnothing )=0$
• Countable additivity: $\mu (\underset{i}{⨄}{A}_i)=\sum \mu (A_i)$

where $\uplus$ is a union of the pairwise disjoint sets

Kinds

• Probability Measure
• Finite Measure
• Sigma-Finite Measure
• Counting measure
• Dirac measure

probability measure $\subset$ finite measure $\subset$ $\sigma$-finite measure

Summary

	$\mu (\varnothing )=0$	$\sigma$-add	$A_i\uparrow \Omega$, $\mu (A_i)<\infty$	$\mu (\Omega )<\infty$	$\mu (\Omega )=1$		monotone	$\sigma$-subadd	conti-below	conti-above
msr	$O$	$O$	$X$	$X$	$X$		$O$	$O$	$O$	$\Delta$
$\sigma$-finite-msr	$O$	$O$	$O$	$X$	$X$		$O$	$O$	$O$	$\Delta$
finite-msr	$O$	$O$	$O$	$O$	$X$		$O$	$O$	$O$	$O$
prob-msr	$O$	$O$	$O$	$O$	$O$		$O$	$O$	$O$	$O$

Notations

• $\sigma$-additive: $\mu (\underset{i=1}{\overset{\infty }{⨄}}{B}_i)={\sum }_{i=1}^{\infty }\mu (B_i)$
• $\sigma$-subadditive: $\mu (\bigcup _{i=1}^{\infty }{A}_i)\le \sum _{i=1}^{\infty }\mu (A_i)$
• monotone: $A\subset B\Rightarrow \mu (A)\le \mu (B)$
• continuous from below: $A_i\uparrow A\Rightarrow \mu (\underset{n\to \infty }{\lim }{A}_i)=\underset{n\to \infty }{\lim }\mu (A_i)$
• continuous from above: $[A_i\downarrow A\text{and}\mu (A_1)<\infty ]\Rightarrow \mu (\underset{n\to \infty }{\lim }{A}_i)=\underset{n\to \infty }{\lim }\mu (A_i)$

Link to original

Measurable Space

Definition

Consider a set $X$ and the sigma-filed defined on the set $X$ $(X,\mathcal{A})$

In contrast to a Measure Space, no Measure is needed for a measurable space

Link to original

Measure Space

Definition

A measure space consists of a Measurable Space together Measure on it

$(X,\mathcal{A},\mu )$ where $X$ is a set, $\mathcal{A}$ is a Sigma-Field defined on the set $X$, and $\mu$ is a Measure.

Kinds

• Probability Space
• Sigma-Finite Measure Space

Link to original

Sigma-Field Generated by A Families of Sets

Definition

$\forall \mathcal{A}\subset 2^{\Omega },\exists !,\sigma (\mathcal{A})$ Intersections of sigma-fields are a sigma-field, and the Intersection of all Sigma-Field containing $\mathcal{A}$ is the smallest Sigma-Field containing $\mathcal{A}$. Thus, there exist the unique smallest Sigma-Field containing every set in the given Family of Sets $\mathcal{A}$.

Link to original

Dynkin's Pi-Lambda Theorem

Definition

$l(\mathcal{P})=\sigma (\mathcal{P})$ where $\mathcal{P}$ is a Pi-System

If $\mathcal{P}$ is a Pi-System, then the small Lambda-System containing $\mathcal{P}$, $l(\mathcal{P})$ is the sigma-field generated by $\mathcal{P}$, $\sigma (\mathcal{P})$.

$\sigma (\mathcal{P})\subset \mathcal{L}$ where $\mathcal{P}$ is a Pi-System and $\mathcal{L}$ is a Lambda-System containing $\mathcal{P}$

If $\mathcal{P}$ is a Pi-System and $\mathcal{L}$ is the Lambda-System containing $\mathcal{P}$, then $\sigma (\mathcal{P})\subset \mathcal{L}$

Proof

If $\mathcal{A}$ is a both Lambda-System and Pi-System, then $\mathcal{A}$ is a Sigma-Field

Since the condition of $\sigma (\mathcal{P})$ is more complex, then $l(\mathcal{P})$, $l(\mathcal{P})\subset \sigma (\mathcal{P})$ is satisfied

The $l(\mathcal{P})$ containing a Pi-System $\mathcal{P}$ is a Pi-System itself. If Lambda-System and Pi-System, then Sigma-Field Therefore Lambda-System $l(\mathcal{P})$ is a Sigma-Field

Since $\sigma (\mathcal{P})$ is the smallest Sigma-Field, $l(\mathcal{P})\supset \sigma (\mathcal{P})$ is satisfied

Summary

	$A\cap B$	$\varnothing$	$A-B$	${\cup }_i{A}_i=\underset{i}{⨄}{B}_i$	$\Omega$	$A^c$	$A\cup B$	$\bigcup _{i=1}^{\infty }{A}_i$	$\underset{i=1}{\overset{\infty }{⨄}}{B}_i$	$\bigcap _{i=1}^{\infty }{A}_i$
$\pi$-system	$O$	$X$	$X$	$X$	$X$	$X$	$X$	$X$	$X$	$X$
$\lambda$-system	$X$	$O$	${\Delta }^{\prime }$	$X$	$O$	$O$	$X$	$X$	$O$	$X$
$\sigma$-field	$O$	$O$	$O$	$O$	$O$	$O$	$O$	$O$	$O$	$O$

Link to original

Extension from Pi-System

Definition

Probability Measure

If $\mathcal{A}$ is a Pi-System on the Probability Space $(\Omega ,\sigma (\mathcal{A}),P)$, then the Probability Measure $P:\sigma (\mathcal{A})\to [0,1]$ is uniquely determined by the probability measure-like function on the Pi-System $P^{\ast }:\mathcal{A}\to [0,1]$

Finite Measure

If $\mathcal{A}$ is a Pi-System containing the universal set $\Omega$ on the finite Measure Space $(\Omega ,\sigma (\mathcal{A}),\mu )$, then the Finite Measure $\mu :\sigma (\mathcal{A})\to [0,\mu (\Omega )<\infty ]$ is uniquely determined by the finite measure-like function on the Pi-System ${\mu }^{\ast }:\mathcal{A}\to [0,\mu (\Omega )<\infty ]$

Sigma-Finite Measure

If $\mathcal{A}$ is a Pi-System on the Sigma-Finite Measure Space $(\Omega ,\sigma (\mathcal{A}),\mu )$, then the Sigma-Finite Measure $\mu :\sigma (\mathcal{A})\to [0,\infty ]$ is uniquely determined by the sigma-finite measure-like function on the Pi-System ${\mu }^{\ast }:\mathcal{A}\to [0,\infty ]$

Facts

If the two measures ${\mu }_1,{\mu }_2$ defined on a Sigma-Field Generated by A Families of Sets $\sigma (\mathcal{A})$ satisfy the following conditions, then they are the same ${\mu }_1={\mu }_2$.

• $\forall A\in \mathcal{A}:{\mu }_1(A)={\mu }_2(A)$
• ${\mu }_1(\Omega )={\mu }_2(\Omega )<\infty$

Link to original

Caratheodory Extension Theorem

Definition

If $\mathcal{A}$ is a Semiring on $\Omega$, and the function (pre-measure) $\tilde{m}:\mathcal{A}\to [0,\infty ]$ satisfies the following conditions, then the function $\tilde{m}$ uniquely extended to the Measure on $(\Omega ,\sigma (\mathcal{A}))$.

Conditions

• $\tilde{m}(\varnothing )=0$
• additivity: $\tilde{m}(\underset{i=1}{\overset{n}{⨄}}{B}_i)={\sum }_{i=1}^n\tilde{m}(B_i)$
• $\sigma$-sub additivity: $\tilde{m}(\bigcup _{i=1}^{\infty }{A}_i)\le {\sum }_{i=1}^{\infty }\tilde{m}(A_i)$
• $\sigma$-finite: $\exists \{A_i{\}}_{i=1}^{\infty }\subset \mathcal{A},\text{s.t.}m(A_i)<\infty ,\bigcup _{i=1}^{\infty }{A}_i=\Omega$

The conditions 1~3: $\tilde{m}$ satisfies measure-like conditions on $\mathcal{A}$ The condition 4: $\tilde{m}$ satisfies Sigma-Finite Measure-like condition on $\mathcal{A}$, it makes the extension result unique.

The set $\mathcal{A}$ should be Semiring. So, the conditions of this theorem are stronger than Extension from Pi-System, but can be applied for every Measure.

Link to original

Usual Topology

Definition

$\mathcal{U}=\left\{O|O={\bigcup }_{i=1}^{\infty }(a_i,b_i)\text{s.t.}(a_i,b_i\in \Omega \text{and}{a}_i\le b_i)\right\}$

$\mathcal{U}$ is a Family of Sets whose elements can be expressed as a countable union of open intervals.

Facts

The usual topological space $(\mathbb{R},\mathcal{U})$ satisfies $\forall o\in O,\exists a,b\in \mathbb{R},o\in (a,b)\subset O$ where $O$ is the set of open intervals

Link to original

Borel Sigma-Field

Definition

$\mathcal{R}=\mathcal{B}(X):=\sigma (\mathcal{T})$

Let $(X,\mathcal{T})$ be a Topological Space, where $X$ is the set and $\mathcal{T}$ is the topology on $X$. The Borel sigma-field $\mathcal{B}(X)$ is the smallest Sigma-Field that contains all open sets in $\mathcal{T}$.

Borel Sigma-Field on the Real Number

$\mathcal{R}=\mathcal{B}(\mathbb{R}):=\sigma (\mathcal{U})$ where $\mathcal{U}$ is Usual Topology on $\mathbb{R}$.

Borel Sigma-Field on the Extended Real Number Space

$\bar{\mathcal{R}}=\mathcal{B}(\bar{\mathbb{R}})$ where $\bar{\mathbb{R}}=\mathbb{R}\cup \{-\infty ,\infty \}$.

Facts

Every element of $\mathcal{B}(\mathbb{R})$ is measurable by Lebesgue Measure

• ${\mathcal{A}}_1:=\{A\subset \mathbb{R}:A\text{ is U-open}\}=\mathcal{U}$
• ${\mathcal{A}}_2:=\{(a,b):a,b\in \mathbb{R},a<b\}$
• ${\mathcal{A}}_3:=\{[a,b):a,b\in \mathbb{R},a<b\}$
• ${\mathcal{A}}_4:=\{(a,b]:a,b\in \mathbb{R},a<b\}$
• ${\mathcal{A}}_5:=\{[a,b]:a,b\in \mathbb{R},a<b\}$
• ${\mathcal{A}}_6:=\{(-\infty ,b):a,b\in \mathbb{R},a<b\}$
• ${\mathcal{A}}_7:=\{(-\infty ,b]:a,b\in \mathbb{R},a<b\}$
• ${\mathcal{A}}_8:=\{(a,\infty ):a,b\in \mathbb{R},a<b\}$
• ${\mathcal{A}}_9:=\{[a,\infty ):a,b\in \mathbb{R},a<b\}$

$\mathcal{R}:=\mathcal{B}(\mathbb{R})=\sigma ({\mathcal{A}}_1)=\sigma ({\mathcal{A}}_2)=\cdots =\sigma ({\mathcal{A}}_9)$ The above sets are all the Borel sigma-field.

Proof Every element of $\sigma (\mathcal{A})$ is generated by the elements of $\mathcal{A}$ and the complementary set and countable union by the conditions of Sigma-Field If $\sigma ({\mathcal{A}}_1)$ can be generated by ${\mathcal{A}}_1$ and the operations, and ${\mathcal{A}}_1$ can be generated by ${\mathcal{A}}_2$ and the operations, then ${\mathcal{A}}_2$ also generate $\sigma ({\mathcal{A}}_1)$ Therefore, $\sigma ({\mathcal{A}}_1)=\sigma ({\mathcal{A}}_2)$

An element of Borel sigma-field $B\in \mathcal{B}(X)$ is called Borel set.

Link to original

Lebesgue Measure

Definition

A Measure on Borel Sigma-Field, which is a unique extension of function $\tilde{m}:\mathcal{A}\to [0,\infty ]$ on a Semiring $\mathcal{A}=\{(a,b]:a,b\in \mathbb{R},a<b\}$ by Caratheodory Extension Theorem

Link to original

Link to original

Probability Theory Note

Random Variable

Definition

$X:\Omega \to \mathbb{R}\text{s.t.}\forall B\in \mathcal{R},\exists \omega \in \Omega ,\text{s.t.}{X}^{-1}(B)=\{\omega :X(\omega )\in B\}\in \mathcal{F}$

A random variable is a function $X:\Omega \to \mathbb{R}$ whose inverse function $X^{-1}(B)$ is $\mathcal{F}$-measurable for the two measurable spaces $(\Omega ,\mathcal{F})$ and $(\mathbb{R},\mathcal{R})$.

The inverse image of an arbitrary Borel set of Codomain $\mathbb{R}$ is an element of sigma field $\mathcal{F}$.

Notations

Consider a probability space $(\Omega ,\mathcal{F},\mathbb{P})$

• Outcomes: $H,T$
• Set of outcomes (Sample space): $\Omega =\{H,T\}$
• Events: $\varnothing ,\{H\},\{T\},\Omega$
• Set of events (Sigma-Field): $\mathcal{F}=\{\varnothing ,\{H\},\{T\},\{H,T\}\}$
• Probabilities: $\mathbb{P}:\mathcal{F}\to [0,1]$
• Random variable: $X:\Omega \to \mathbb{R}$

For a random variable $X$ on a Probability Space $(\Omega ,\mathcal{F},P)$

• $X\sim {\mu }_X$ if and only if the Distribution of $X$ is ${\mu }_X$
• $X\sim F_X$ if and only if the Distribution Function of $X$ is $F_X$

For a random variable $X$ on Probability Space $({\Omega }_X,{\mathcal{F}}_X,P_X)$ and another random variable $Y$ on Probability Space $({\Omega }_Y,{\mathcal{F}}_Y,P_Y)$

• $X\stackrel{d}{=}Y⟺\forall B\in \mathcal{R}:{\mu }_X(B)={\mu }_Y(B)$
• $X\stackrel{d}{=}Y⟺\forall k\in \mathbb{R}:F_X(k)=F_Y(k)$
• $X\stackrel{d}{=}Y⟺\forall k\in \mathbb{R}:P_X(X\le k)=P_Y(Y\le k)$

Link to original

Distribution

Definition

${\mu }_X:=P\circ X^{-1}:\mathcal{R}\to [0,1]$ where $\mathcal{R}$ is a Borel Sigma-Field

A distribution ${\mu }_X$ is a function for the Random Variable $X$ on Probability Space $(\Omega ,\mathcal{F},P)$

Facts

${\mu }_X$ is a Probability Measure on $(\mathbb{R},\mathcal{R})$, so $(\mathbb{R},\mathcal{R},{\mu }_X)$ is a Probability Space. The Probability Space $(\mathbb{R},\mathcal{R},{\mu }_X)$ is called the Probability Space induced by $X$

Link to original

Distribution Function

Definition

$F_X(x):\mathbb{R}\to [0,1]:={\mu }_X((-\infty ,x])=P(X\le x)$

A distribution function $F_X$ is a function for the Random Variable on Probability Space $(\Omega ,\mathcal{F},P)$

Facts

$(\Omega ,\mathcal{F},P)\sim (\mathbb{R},\mathcal{R},{\mu }_X)$

Proof By definition, $F_X(x)={\mu }_X((-\infty ,x])$ So, defining $\forall x\in \mathbb{R},F_X(x)$ is the Equivalence Relation to defining $\mathcal{A}=\{(-\infty ,x]:x\in \mathbb{R}\},{\mu }_X:\mathcal{A}\to [0,1]$

Since $\mathcal{A}$ is a Pi-System, ${\mu }_X:\sigma (\mathcal{R})=\mathcal{R}\to [0,1]$ is uniquely determined by ${\mu }_X:\mathcal{A}\to [0,1]$ by Extension from Pi-System

Now, $(\mathbb{R},\mathcal{R},{\mu }_X)$ is a Measurable Space induced by $X$. Therefore, defining ${\mu }_X$ on $(\mathbb{R},\mathcal{R})$ is equivalent to the defining $P$ on $(\Omega ,\mathcal{F})$

Distribution function(CDF) $F:\mathbb{R}\to [0,1]$ has the following properties

• Monotonic increasing: $\forall a,b,\in \mathbb{R},a<b\Rightarrow F(a)\le F(b)$
• $\underset{x\to -\infty }{\lim }F(x)=0$
• $\underset{x\to \infty }{\lim }F(x)=1$
• Right-continuous: $\underset{x\to x_0^{+}}{\lim }F(x)=F(x_0)$

If a function $F:\mathbb{R}\to [0,1]$ satisfies the following properties, then $F$ is a distribution function(CDF) of some Random Variable $X$

• Monotonic increasing: $\forall a,b,\in \mathbb{R},a<b\Rightarrow F(a)\le F(b)$
• $\underset{x\to -\infty }{\lim }F(x)=0$
• $\underset{x\to \infty }{\lim }F(x)=1$
• Right-continuous: $\underset{x\to x_0^{+}}{\lim }F(x)=F(x_0)$

$\forall a,b\in \mathbb{R},a<b\Rightarrow (P(a<X\le b)=F_X(b)-F_X(a))$

Link to original

Transclude of Density-Function

Extended Real Number Space

Definition

$\bar{\mathbb{R}}=\mathbb{R}\cup \{-\infty \}\cup \{\infty \}$

Link to original

Measurable Function

Definition

$f:\Omega \to S\text{s.t.}\forall B\in \mathcal{S},f^{-1}(B)=\{\omega \in \Omega |f(\omega )\in B\}\in \mathcal{F}$

A measurable function is a function $f:\Omega \to S$ whose inverse function $X^{-1}(B)$ is $\mathcal{F}$-measurable the two measurable spaces $(\Omega ,\mathcal{F}),(S,\mathcal{S})$.

Other Expressions

$f:\Omega \to S$ is a measurable function on the two measurable spaces $(\Omega ,\mathcal{F}),(S,\mathcal{S})$ $f:(\Omega ,\mathcal{F})\to (S,\mathcal{S})$ $f$ is $\mathcal{F}-\mathcal{S}$ measurable $f\in \mathcal{F}$ (only if $(S,\mathcal{S})=(\mathbb{R},\mathcal{R})$) $f$ is a $S$-valued measurable

Examples

When $(S,\mathcal{S})=(\mathbb{R},\mathcal{R})$, the measurable function is called a Random Variable. When $(S,\mathcal{S})=({\mathbb{R}}^d,{\mathcal{R}}^d)$, the measurable function is called a Random Vector.

Facts

For the $\mathcal{F}$-measurable set $A\in \mathcal{F}$ on the two measurable spaces $(\Omega ,\mathcal{F}),(\mathbb{R},\mathcal{R})$ A function $f:\Omega \to \mathbb{R}$ defined as $f(\omega )=\{\begin{array}{ll}1& \omega \in A\\ 0& \omega \notin A\end{array}$ is a $\mathcal{F}-\mathcal{R}$ measurable function

If $f$ is a $\mathbb{R}$-valued measurable function on the two measurable spaces $(\Omega ,\mathcal{F}),(\mathbb{R},\mathcal{R})$, then $\forall \alpha \in \mathbb{R},\alpha f(\omega )$ also $\mathbb{R}$-valued measurable. Scaling of function does not change the Domain, only scale the Image

If $f:(\Omega ,\mathcal{F})\to (S,\mathcal{S})$ and $g:(S,\mathcal{S})\to (T,\mathcal{T})$ on the three measurable spaces $(\Omega ,\mathcal{F}),(S,\mathcal{S}),(T,\mathcal{T})$, then $g\circ f:(\Omega ,\mathcal{F})\to (T,\mathcal{T})$

If a function $f:\mathbb{R}\to \mathbb{R}$ is continuous, is measurable.

If two functions $f,g$ are $\mathbb{R}$-valued measurable on the two measurable spaces $(\Omega ,\mathcal{F}),(\mathbb{R},\mathcal{R})$, then following are all $\mathbb{R}$-valued measurable

• $h:=f+g$
• $h:=f-g$
• $h:=\max (f,g)$, where $h:\omega \mapsto \max (f(\omega ),g(\omega ))$
• $h:=\min (f,g)$, where $h:\omega \mapsto \min (f(\omega ),g(\omega ))$

If every element of a function sequence $\{f_n:n\in \mathbb{N}\}$ is $\mathbb{R}$-valued measurable, then the following are all $\mathbb{R}$-valued measurable.

• $h:=\inf f_n$, where $h:\omega \mapsto \inf \{f_1(\omega ),f_2(\omega ),\dots \}$
• $h:=\sup f_n$, where $h:\omega \mapsto \sup \{f_1(\omega ),f_2(\omega ),\dots \}$

Link to original

Lebesgue Integral Note

Almost Everywhere

Definition

The Propositional Function is satisfied for every point except where Lebesgue Measure is 0

Link to original

Lebesgue Integral

Definition

Riemann integral and Lebesgue integral

Consider a measure spaces $(\Omega ,\mathcal{F},\mu )$ and a Measure Space $(\overline{\mathbb{R}},\mathcal{R})$, where $\mathcal{R}$ is the Borel Sigma-Field on $\overline{\mathbb{R}}$, and a Measurable Function $f:(\Omega ,\mathcal{F})\to (\overline{\mathbb{R}},\mathcal{R})$. Then the Lebesgue integral of the function $f$ with respect to $\mu$ is denoted as:

${\int }_{\Omega }fd\mu ={\int }_{\Omega }f(x)d\mu (x)={\int }_{\Omega }f(x)\mu (dx)$

Approximation of Lebesgue Integral with Finite Sum

$\int fd\lambda \approx \sum _{i=1}^N{a}_n\lambda (A_n)$ where $A_n:=\{x|f(x)=a_n\}$ or $A_n=f^{-1}(\{a_n\})$.

To compute the Riemann Integral of $f$, one partitions the domain $[a,b]$ into sub-intervals, while in the Lebesgue integral, one partitions the image of $f$.

Link to original

Simple Function

Definition

$f(\omega )=\sum _{i=1}^n{\alpha }_i{1}_{A_i}(\omega )$ where $A_i\in \mathcal{F}$ are disjoint measurable sets, $a_i$ are the sequence of real or complex numbers, and $1_A$ is a Indicator Function

Finite linear combination of indicator functions of measurable sets

Facts

Simple function $f$ is $\mathcal{F}-\mathcal{R}$ measurable map

A Measurable Function $f$ with a finite Image is a Simple Function

Link to original

Lebesgue Integral of Simple Function

Definition

Consider a measure spaces $(\Omega ,\mathcal{F},\mu )$ and a Measure Space $(\overline{\mathbb{R}},\mathcal{R})$, and a positive $\mathcal{F}-\mathcal{R}$ measurable Simple Function, $f:(\Omega ,\mathcal{F})\to (\overline{\mathbb{R}},\mathcal{R})$, with Codomain $R:=\{r_1,r_2,\dots ,r_n\}$ that have corresponding preimages $f^{-1}(r_i)=A_i\in \mathcal{F}$. Then the Lebesgue integral of the function $f$ with respect to $\mu$ is defined as ${\int }_{\Omega }fd\mu ={\sum }_{i=1}^n{r}_i\mu (A_i)$

Facts

Consider a measure spaces $(\Omega ,\mathcal{F},\mu )$ and a Measure Space $(\overline{\mathbb{R}},\mathcal{R})$, and a positive $\mathcal{F}-\mathcal{R}$ measurable Simple Function, $f:(\Omega ,\mathcal{F})\to (\overline{\mathbb{R}},\mathcal{R})$, with Codomain $R:=\{r_1,r_2,\dots ,r_n\}$ that have corresponding preimages $f^{-1}(r_i)=A_i\in \mathcal{F}$. Then, $f={\sum }_{i=1}^m{\alpha }_i{1}_{A_i}={\sum }_{j=1}^n{\beta }_j{1}_{B_j}⟹\int fd\mu ={\sum }_{i=1}^n{\alpha }_i\mu (A_i)={\sum }_{i=1}^n{\beta }_i\mu (B_i)$

Link to original

Lebesgue Integral of Non-Negative Function

Definition

Consider a measure spaces $(\Omega ,\mathcal{F},\mu )$ and a Measure Space $(\overline{\mathbb{R}},\mathcal{R})$ and a positive Measurable Function $f:(\Omega ,\mathcal{F})\to (\overline{\mathbb{R}},\mathcal{R})$. Then the Lebesgue integral of the function $f$ with respect to $\mu$ is defined as ${\int }_{\Omega }fd\mu :=\sup \left\{{\int }_{\Omega }\phi d\mu \right|0\le \phi \le f,\text{a.e. w.r.t.}\mu \}$ where each $\phi :(\Omega ,\mathcal{F})\to (\mathbb{R},\mathcal{R})$ is a Simple Function.

Facts

$\int fd\mu$ can be infinity

Link to original

Lebesgue Integral of Measurable Function

Definition

Consider a measure spaces $(\Omega ,\mathcal{F},\mu )$ and a Measure Space $(\overline{\mathbb{R}},\mathcal{R})$ and a Measurable Function $f:(\Omega ,\mathcal{F})\to (\overline{\mathbb{R}},\mathcal{R})$. Then the Lebesgue integral of the function $f$ with respect to $\mu$ is defined as ${\int }_{\Omega }fd\mu :={\int }_{\Omega }{f}^{+}d\mu -{\int }_{\Omega }{f}^{-}d\mu$ where $f^{+}:=\max (0,f)$ and $f^{-}:=\max (0,-f)$ for the function $f$.

Existence of Integral

$\int fd\mu$ exists

• $\int f^{+}d\mu <\infty$ and $\int f^{-}d\mu <\infty$ ($f$ is integrable w.r.t $\mu$) $\int fd\mu =\int f^{+}d\mu -\int f^{-}d\mu$
• $\int f^{+}d\mu =\infty$ and $\int f^{-}d\mu <\infty$ $\int fd\mu =\infty$
• $\int f^{+}d\mu <\infty$ and $\int f^{-}d\mu =\infty$ $\int fd\mu =-\infty$

$\int fd\mu$ not exists

• $\int f^{+}d\mu =\infty$ and $\int f^{-}d\mu =\infty$ $\int fd\mu$ is not defined

Notations

For a Density Function $F_X={\mu }_X((-\infty ,x])$ on the Measurable Space $(\mathbb{R},\mathcal{R},{\mu }_X)$ $\int gd{\mu }_X=\int gd{F}_X=\int g(x)d{F}_X(x)$

For a Density Function $F_X={\mu }_X((-\infty ,x])$ that has Density Function $f_X(x)$ on the Measurable Space $(\mathbb{R},\mathcal{R},{\mu }_X)$ $\int g(x)d{F}_X(x)=\int g(x)f_X(x)dx$

Consider a Measure Space $(\Omega ,\mathcal{F},\#)$ where $\Omega$ is a countable set, $\mathcal{F}=2^{\Omega }$ is a Sigma-Field defined on the set $\Omega$, and $\#$ is a Counting measure on $\mathcal{F}$. Then $\int fd\#=\sum _{i\in \Omega }f(i)$

Facts

For the functions $f^{+}:=\max (0,f)$ and $f^{-}:=\max (0,-f)$ of $f$,

• $f^{+},f^{-}$ are non-negative
• $|f|=f^{+}+f^{-}$
• $f=f^{+}-f^{-}$
• If $f$ is measurable, then $f^{+}$ and $f^{-}$ are also measurable.

Link to original

Link to original

Cantor Set

Definition

$\mathcal{C}:=\underset{n\to \infty }{\lim }{C}_n={\bigcap }_{n=0}^{\infty }{C}_n$ where $C_0:=[0,1]$ and $C_n:=\frac{C_{n-1}}{3}\cup \left(\frac{2}{3}+\frac{C_{n-1}}{3}\right)$

Cantor set is created by iteratively deleting the open middle third from a set of line segments.

Facts

The elements of $C$ are numbers consisting of 0 or 2 in ternary

$|\mathcal{C}|=|[0,1]|=2^{\aleph }$ The Cardinality of $\mathcal{C}$ is equal to the interval $[0,1]$

$\lambda (\mathcal{C})=0$ $\mathcal{C}$ is a Lesbesgue-measurable set, and its Lebesgue Measure is $0$

Link to original

Cantor Function

Definition

$F(x)=\underset{n\to \infty }{\lim }{F}_n(x)$

A function $F$ that has positive derivative on Cantor Set $\mathcal{C}$ and zero derivative on $[0,1]-\mathcal{C}$

Facts

Consider the Cantor function $F$

• $F$ is a Constant Function on $[0,1]-\mathcal{C}$
• $F$ has zero derivative Almost Everywhere
• $F$ is a non-negative function

Consider the Cantor function $F$

• $F$ is monotonically increasing
• $F(0)=0$
• $F(1)=1$
• $F$ is continuous everywhere

Thus by the property of distribution function, there exists a continuous Random Variable corresponding to $F$.

$F$ has a derivative at Almost Everywhere, but the integral of the derivative and $F$ is not the same.

Link to original

Absolute Continuity

Definition

$\mu \ll \lambda$ Consider measures $\mu ,\lambda$ on a Measurable Space $(\Omega ,\mathcal{F})$. The measure $\mu$ said to be absolutely continuous with respect to $\lambda$, or $\mu$ is dominated by $\lambda$, denoted by $\mu \ll \lambda$, If $\forall S\in \mathcal{F},\lambda (S)=0\Rightarrow \mu (S)=0$

Summary

Comparison of conditions

Function	Derivative	Radon–Nikodym Derivative
Continuous Function	X	X (Cantor Function)
Absolute continuous function	X	O
Differentiable function	O	O

• Differentiable function $\subset$ Absolute continuous function $\subset$ Continuous function

Facts

Continuously differentiable $\subset$ Lipschitz continuous $\subset$ $\alpha -$Holder Continuous $\subset$ Uniformly Continuous $\subset$ Continuous Lipschitz continuous Absolute continuous $\subset$ Uniformly Continuous $\subset$ Continuous where $0<0\le 1$

Link to original

Link to original

Radon–Nikodym Theorem

Definition

Consider sigma-finite measures $\mu ,\lambda$ on a Measurable Space $(S,\mathcal{S})$. If $\mu \ll \lambda$ ($\mu$ is absolutely continuous with respect to $\lambda$), then there exists a $\mathcal{S}$-Measurable Function $f:X\to {\mathbb{R}}_0^{+}$ such that $\forall B\in \mathcal{S},\mu (B)={\int }_{B}fd\lambda$ almost uniquely with respect to $\lambda$. Where the function $f:(S,\mathcal{R})\to ({\mathbb{R}}^{+},{\mathcal{R}}^{+})$ is called a Radon-Nikodym derivative of $\mu$ w.r.t. $\lambda$ $\mu \ll \lambda \Rightarrow \exists !f=\frac{d\mu }{d\lambda }\text{s.t.}\forall B\in \mathcal{S},\mu (B)={\int }_{B}fd\lambda$

Radon–Nikodym Derivative

$f=\frac{d\mu }{d\lambda }$ Consider sigma-finite measures $\mu ,\lambda$ on a Measurable Space $(S,\mathcal{S})$. A Radon–Nikodym derivative $f=\frac{d\mu }{d\lambda }$ is a function satisfying $\forall B\in \mathcal{S},\mu (B)={\int }_{B}fd\lambda$.

Creating New Measure

Given a measure space $(S,\mathcal{S},\lambda )$ and a non-negative Measurable Function $f:S\to {\mathbb{R}}_0^{+}$, we can define a new measure $\mu$ by the relation $\forall B\in S,\mu (B)={\int }_{B}fd\lambda$ The measure $\mu$ is absolutely continuous with respect to $\lambda$ ($\mu \ll \lambda$) and $f$ be a Radon-Nikodym derivative of $\mu$ with respect to $\lambda$.

Examples

Distribution Case

Consider a distribution $F_X:\mathbb{R}\to [0,1]$ and a Lebesgue Measure $\lambda$, and a Measurable Space $(\mathbb{R},\mathcal{R})$ (Borel Sigma-Field on Real Number). If ${\mu }_X\ll \lambda$, then there exists a Density Function $f_X$ of the Distribution Function $F_X$ ${\mu }_X\ll \lambda \Rightarrow \exists !f_X=\frac{d{\mu }_X}{d\lambda }\text{s.t.}{F}_X={\int }_{(-\infty ,x]}{f}_{X}d\lambda$

Facts

If another function $g$ have the same properties, then $f=g$ at Almost Everywhere

Link to original

Generalized Density Function

category	measurable space	distribution	measure	description
Continuous	$(\mathbb{R},\mathcal{R})$	${\mu }_X$	$\lambda$
Discrete	$(S,\mathcal{S})$	${\tilde{\mu }}_X:\mathcal{S}\to [0,1]$	$\#$: Counting measure	$\#$ is not sigma-finite on $(\mathbb{R},\mathcal{R})$
Discrete	$(\mathbb{R},\mathcal{R})$	${\mu }_X$	${\#}_X$	${\#}_X$ is sigma-finite on $(\mathbb{R},\mathcal{R})$

Where:

• $S$ is a support of a discrete random variable
• $\mathcal{S}=2^S$.
• ${\#}_X:={\sum }_{i\in X}{\delta }_i$: only counts the number of elements in the support of a discrete random variable $X$. It can be expressed as the sum of Dirac measure

Link to original

Lebesgue Decomposition Theorem

Definition

A Distribution $F$ can be decomposed as the following $F=F_{ac}+F_{pp}+F_{sing}$ Where:

• $F_{ac}$: absolutely continuous part (absolutely continuous w.r.t $\lambda$)
• $F_{pp}$: pure point part: (absolutely continuous w.r.t ${\#}_X$)
• $F_{sing}$: singular continuous part (e.g. cantor distribution)

Facts

$F_{pp}$ has countable discontinuous points

If a Distribution Function $F=F_{ac}$, then there exists corresponding continuous Random Variable $X$ and PDF

If a Distribution Function $F=F_{pp}$, then there exists corresponding discrete Random Variable $X$ and generalized PDF or PMF

If a Distribution Function $F=F_{ac}+F_{pp}$, then there exists corresponding mixed Random Variable $X$ and generalized PDF

Link to original

Expected Value

Definition

$E(X)={\int }_{\Omega }XdP={\int }_{\mathbb{R}}xd{\mu }_X={\int }_{\mathbb{R}}xd{F}_X$ The expected value of the Random Variable $X$ on Probability Space $(\Omega ,\mathcal{F},P)$

Continuous

$E(X)={\int }_{-\infty }^{\infty }x{f}_X(x)dx$ where $f_X(x)$ is a PDF of Random Variable $X$

The expected value of the Random Variable $X$ when $F_X$ satisfies absolute continuous over $\lambda$, ${\mu }_X<<\lambda$

Discrete

$E(X)={\sum }_{x}x{p}_X(x)$ where $p_X(x)$ is a PMF of Random Variable $X$

The expected value of the Random Variable $X$ when $F_X$ satisfies absolute continuous over $\#$, ${\mu }_X<<{\#}_X$ In other words, $F_X$ has at most countable jumps.

Expected Value of a Function

$E(g(x))={\int }_{\mathbb{R}}g(x)d{\mu }_X$

Continuous

$E(g(x))={\int }_{\mathbb{R}}g(x)f_X(x)dx$

Discrete

$E(g(x))=\sum _{x\in S_X}g(x)p_X(x)$

Properties

Linearity

Random Variables

If $\exists k_{i}E[X_i]$, then $E\left[{\sum }_{i=1}^n{k}_i{X}_i\right]={\sum }_{i=1}^n{k}_{i}E(X_i)$

Matrix of Random Variables

Let $W_1,W_2$ be a $m\times n$ matrices of random variables, $A_1,A_2$ be $k\times m$ matrices of constants, and $B$ a $n\times l$ matrix of constant. Then, $E[A_1{W}_1+A_2{W}_2]=A_{1}E[W_1]+A_{2}E[W_2]$

$E[A_1{W}_{1}B]=A_{1}E[W_1]B$

Notations

Expression	Discrete	Continuous
Expression for the event $2^{\Omega }$ and the probability $P$	${\sum }_{\omega \in \Omega }X(\omega )\cdot P(\omega )$	${\int }_{\Omega }X(\omega )\cdot dP(\omega )$
Expression for the measurable space $(\mathbb{R},\mathcal{R})$ and the distribution ${\mu }_X$	${\sum }_{x\in \mathbb{R}}x\cdot f(x)$	${\int }_{\mathbb{R}}f(x)\lambda (x)={\int }_{\mathbb{R}}\frac{d{\mu }_X}{d\lambda }\lambda (x)={\int }_{\mathbb{R}}d{\mu }_X={\int }_{\mathbb{R}}d{F}_X$

Link to original

Link to original