Derivatives of Matrix

Definition

Differentiation by vector

$\frac{\partial }{\partial {\mathbf{x}}^{⊺}}:=\left[\frac{\partial }{\partial x_1},\frac{\partial }{\partial x_2},\dots ,\frac{\partial }{\partial x_n}\right]$ where $\mathbf{x}=[x_1,x_2,\dots ,x_n]$

Differentiation by Matrix

$\frac{\partial }{\partial X}:=\left[\begin{array}{ccc}\frac{\partial }{\partial x_{11}}& \dots & \frac{\partial }{\partial x_{1p}}\\ ⋮& \ddots & ⋮\\ \frac{\partial }{\partial x_{n1}}& \dots & \frac{\partial }{\partial x_{np}}\end{array}\right]$ where $X=\left[\begin{array}{ccc}{x}_{11}& \dots & x_{1p}\\ ⋮& \ddots & ⋮\\ x_{n1}& \dots & x_{np}\end{array}\right]$

Calculation

Let $f$ be a vector function, $\mathbf{x}$ be an $n$-dimensional vector, then $\frac{\partial f}{\partial {\mathbf{x}}^{⊺}}=\left[\frac{\partial f}{\partial x_1},\frac{\partial f}{\partial x_2},\dots ,\frac{\partial f}{\partial x_n}\right]$ By the definition of derivative, we hold $\mathbf{df}=f(\mathbf{x}+d\mathbf{x})-f(\mathbf{x})=\frac{\partial f}{\partial x_1}d{x}_1+\frac{\partial f}{\partial x_2}d{x}_2+\cdots +\frac{\partial f}{\partial x_n}d{x}_n=\frac{\partial f}{\partial {\mathbf{x}}^{⊺}}d\mathbf{x}$ where $d\mathbf{x}=[d{x}_1,d{x}_2{]}^{⊺}$

Examples

Let $\mathbf{x},\mathbf{y}$ be $n$-dimensional vectors, then $\frac{\partial }{\partial \mathbf{x}}({\mathbf{x}}^{⊺}\mathbf{y})=\frac{\partial }{\partial \mathbf{x}}({\mathbf{y}}^{⊺}\mathbf{x})=\mathbf{y}$

Let $\mathbf{x}$ be an $n$-dimensional vector and $A$ be an $n\times n$ matrix, then $\frac{\partial }{\partial \mathbf{x}}({\mathbf{x}}^{⊺}A)=A,\frac{\partial }{\partial \mathbf{x}}(A\mathbf{x})=A^{⊺}$

Let $\mathbf{x}$ be an $n$-dimensional vector and $A$ be an $n\times n$ Symmetric Matrix, then the derivative of quadratic form is $\frac{\partial }{\partial \mathbf{x}}({\mathbf{x}}^{⊺}A\mathbf{x})=A\mathbf{x}+A^{⊺}\mathbf{x}=2A\mathbf{x}$