Covariance Matrix

Definition

Variance-Covariance Matrix

Let $\mathbf{X}=(X_1,X_2,\dots ,X_n)$ be an $n$ dimensional Random Vector with finite variance $Var(X_i)<\infty$, and $\mathbf{\mu }=E[\mathbf{X}]$, then the variance-covariance matrix of $\mathbf{X}$ is $\mathrm{Var}(\mathbf{X})=\mathrm{Cov}(\mathbf{X},\mathbf{X})=E[(\mathbf{X}-\mathbf{\mu })(\mathbf{X}-\mathbf{\mu }{)}^{⊺}]=E(X{X}^{⊺})-E(X)E(X{)}^{⊺}$

Cross-Covariance Matrix

Let $\mathbf{X}=(X_1,X_2,\dots ,X_n),\mathbf{Y}=(Y_1,Y_2,\dots ,Y_n)$ be $n$ dimensional random vectors with finite variance $Var(X_i)<\infty ,Var(Y_i)<\infty$, and ${\mathbf{\mu }}_{\mathbf{X}}=E[\mathbf{X}],{\mathbf{\mu }}_{\mathbf{Y}}=E[\mathbf{Y}]$, then the cross-covariance matrix of $\mathbf{X},\mathbf{Y}$ is $\mathrm{Cov}(\mathbf{X},\mathbf{Y})=E[(\mathbf{X}-{\mathbf{\mu }}_{\mathbf{X}})(\mathbf{Y}-{\mathbf{\mu }}_{\mathbf{Y}}{)}^{⊺}]=[\mathrm{Cov}(X_i,Y_i)]=E(X{Y}^{⊺})-{\mu }_X{\mu }_Y^{⊺}$

Properties

Let $\mathbf{X}$ be a Random Vector with finite variance $Var(X_i)<\infty$, and $A$ be a matrix of constants, then

$\mathrm{Var}(A\mathbf{X})=A\mathrm{Var}(\mathbf{X})A^{⊺}$

Let $X$ and $Y$ be random vectors. $\mathrm{Cov}(AX,BY)=A\mathrm{Cov}(X,Y)B^{⊺}$ where $A$ and $B$ are matrices of constants.

Facts

Every variance-covariance matrix is Positive Semi-Definite Matrix

Let $X$ be a Random Vector such that no element of $X$ is a linear combination of the remaining elements. $\nexists \mathbf{a},b\text{s.t.}\mathbf{a}X=b$ Then, the Variance-Covariance Matrix of the random vector is Positive-Definite Matrix. $\mathrm{Var}(X)\succ 0$