Markov Chain

Definition

$(S,P)$ A Markov chain is a Stochastic Process $\{X_t\}$ that satisfies Markov Property. It consists of a set of states $S$ and a Transition Probability Matrix $P$

Summary

Conditions and Properties of Markov Chain (Finite States)

Example	Finite	IRR	Aperiodicity	$\exists !{\mathbf{p}}^{\star }$^[a unique Limiting Distribution]	$\exists !\mathbf{\pi }$^[a unique Stationary Distribution]	$\bar{\mathbf{\pi }}\to \mathbf{\pi }$^[Ergodic Theorem]	$\bar{\mathbf{\pi }}\to \mathbf{\pi }={\mathbf{p}}^{\star }$^[Ergodicity]
	O	X	X	X	X	X	X
Identity/Block matrix	O	X	O	X	X	X	X
Exchange matrix	O	O	X	X	O	O	X
Ergodic Markov Chain	O	O	O	O	O	O	O

flowchart LR
  mc[HMC] --> finite
  
  subgraph finiteness
    finite([finite])
  end
  
  finite --> finite_mc
  finite_mc --> irreducible
  finite_mc --> reducible
  
  subgraph Irreducibility
    irreducible([irreducible])
    reducible([reducible])
  end
  irreducible --> irr_mc[Irreducible MC]
  reducible -->|divide|irr_mc

  irr_mc ---|holds|egt([Ergodic Theorem])
  irr_mc --> aperiodic
  
  subgraph Aperiodicity
    aperiodic([aperiodic])
  end
  
  aperiodic --> ergmc[Ergodic MC]

Conditions and Properties of Markov Chain (Infinite States)

IRR	Nature	Aperiodicity	$\exists !{\mathbf{p}}^{\star }$^[a unique Limiting Distribution]	$\exists !\mathbf{\pi }$^[a unique Stationary Distribution]	$\bar{\mathbf{\pi }}\to \mathbf{\pi }$^[Ergodic Theorem]	$\bar{\mathbf{\pi }}\to \mathbf{\pi }={\mathbf{p}}^{\star }$^[Ergodicity]
O	Transient	can’t define	X	X	X	X
O	Null recurrent	no matter	X	X	X	X
O	Positive recurrent	X	X	O	O	X
O	Positive recurrent	O	O	O	O	O

flowchart LR
  mc[HMC]
  mc --> irreducible
  mc --> reducible
  
  subgraph Irreducibility
    irreducible([irreducible])
    reducible([reducible])
  end
  
  irr_mc[Irreducible MC]
  irreducible --> irr_mc
  reducible -->|divide|irr_mc
  
  irr_mc --> pr
  irr_mc --> nr
  irr_mc --> tr
  
  subgraph Recurrence
    pr([positive recurrent])
    nr([null recurrent])
    tr([transient])
  end
    
  pr_mc[Recurrent MC]
  pr --> pr_mc
  pr_mc --> aperiodic

  egt([Ergodic Theorem])
  pr_mc ---|holds|egt

  subgraph Aperiodicity
    aperiodic([aperiodic])
  end
  
  aperiodic --> ergmc[Ergodic MC]