Bayes Theorem

Definition

Discrete Case

$P(C_j|C)=\frac{P(C|C_j)P(C_j)}{P(C)}=\frac{P(C|C_j)P(C_j)}{\sum _{i=1}^{k}P(C|C_i)P(C_i)},j=1,\dots ,k$ where $\sum _{i=1}^{k}P(C_i)P(C|C_i)=P(C)$ by Law of Total Probability

$P(C_j)$ is called a prior probability, and $P(C_j|C)$ is called a posterior probability

Continuous Case

$f(\theta |x;\gamma )=\frac{f(x|\theta )f(\theta ;\gamma )}{f(x)}=\frac{f(x|\theta )f(\theta ;\gamma )}{\int f(x|\theta )f(\theta ;\gamma )d\theta }\propto f(x|\theta )f(\theta ;\gamma )$ where $\theta$ is not a constant, but an unknown parameter follows a certain distribution with a parameter $\gamma$.

$f(\theta ;\gamma )$ is called a prior probability, $f(x|\theta )$ is called a likelihood, $f(x)$ is called an evidence or marginal likelihood, and $f(\theta |x;\gamma )$ is called a posterior probability

Parameter-Centric Notation

$p(\theta |x)=\frac{p(x|\theta )\pi (\theta )}{p(x)}$

Examples

Consider a random variable follows Binomial Distribution $X\sim B(n,\theta )$ and a prior distribution follows Beta Distribution $\pi (\theta ;\gamma )\sim \mathrm{Beta}(\alpha ,\beta )$ where $\gamma =(\alpha ,\beta )$.

The PDFs are defined as $p(x|\theta )=(\genfrac{}{}{0ex}{}{n}{x}){\theta }^x(1-\theta {)}^{n-x}$ and $\pi (\theta ;\gamma )=\frac{\Gamma (\alpha +\beta )}{\Gamma (\alpha )\Gamma (\beta )}{\theta }^{\alpha -1}(1-\theta {)}^{\beta -1}$. Then, by Bayes theorem, $p(\theta |x)\propto p(x|\theta )\pi (\theta ;\gamma )\propto {\theta }^{x+\alpha -1}(1-{\theta }^{n-x+\beta -1})\sim \mathrm{Beta}(x+\alpha ,n-x+\beta )$ Under Squared Error Loss, the Bayes Estimator is a mean of the posterior distribution. ${\hat{\delta }}_{\text{Bayes}}=\frac{x+\alpha }{n+\alpha +\beta }$