Gradient

Definition

The unique vector $\nabla f$ satisfying following relation. $df(\cdot )=⟨\cdot ,\nabla f⟩$ The existence and uniqueness are guaranteed by Riesz representation theorem

Euclidean Space

Let $f:D\subset {\mathbb{R}}^n\to \mathbb{R},f(x_1,x_2,\dots ,x_n)$ be a function defined for each $\mathbf{p}\in D$, and suppose that its partial derivatives are exist and continuous, then the gradient of the function at the point $\mathbf{p}$ is defined as $\nabla f(\mathbf{p}):=\left(\frac{\partial f}{\partial x_1}(\mathbf{p}),\frac{\partial f}{\partial x_2}(\mathbf{p}),\dots ,\frac{\partial f}{\partial x_n}(\mathbf{p})\right)$

Properties

Operations

Suppose $f,g:D\subset {\mathbb{R}}^n\to \mathbb{R}$. Then, the following are satisfied

• $\nabla (f+g)=\nabla f+\nabla g$
• $\nabla (f-g)=\nabla f-\nabla g$
• $\nabla (kf)=k\nabla f$ where $k\in \mathbb{R}$
• $\nabla (fg)=f\nabla g+g\nabla f$
• $\nabla \left(\frac{f}{g}\right)=\frac{g\nabla f-f\nabla g}{g^2}$

Composition

Suppose differentiable functions $f:D\subset {\mathbb{R}}^n\to {\mathbb{R}}^m$ and $g:D^{\prime }\subset {\mathbb{R}}^m\to {\mathbb{R}}^k$ and assume that for each $\mathbf{x}\in D$, $f(\mathbf{x})\in D^{\prime }$. Then $g\circ f$ is differentiable and the gradient of the composite function is defined as $\nabla (g\circ f)(\mathbf{x})=\nabla g(f(\mathbf{x}))\cdot \nabla f(\mathbf{x})$ here $\nabla g(f(\mathbf{x}))$ is $k\times m$, and $\nabla f(\mathbf{x})$ is $m\times n$ Jacobian matrices, and the result $\nabla (g\circ f)$ is $k\times n$ matrix.

Facts

For a unit directional vector $\mathbf{v}$, Directional Derivative and gradient have the following relation ${\nabla }_{\mathbf{v}}f=\nabla f\cdot \mathbf{v}=||\nabla f||\cos (\theta )$ where $\theta$ is the angle between the vectors $\mathbf{v}$ and $\nabla f$.

So, the function $f$ increases most rapidly when $\theta =0$. In other words, when $\mathbf{v}//\nabla f$. Thus, the gradient is the most rapidly increasing direction of the function $f$. In the same logic, $f$ decreases most rapidly in the direction of $-\nabla f$, and the direction $\mathbf{u}$ orthogonal to a gradient is a direction of non-change in $f$.

Orthogonal Gradient Theorem