Method of Lagrange Multipliers

Definition

$\mathcal{L}(x,\lambda ):=f(x)+\lambda g(x)$

The method of Lagrange multipliers is a method for finding the local maxima and minima of a function subject to equality constraints

Solution

Given general problem $\underset{x}{\min }f(x)\text{subject to}g(x)=0$

In order to find the maximum or minimum of a function $f(x)$ subject to the equality constraint $g(x)=0$, find the stationary point of $\mathcal{L}(x,\lambda )$ considered as a function of $x$ and the Lagrange multiplier $\lambda$. In other words, all partial derivatives should be zero. ${\nabla }_x\mathcal{L}(x,\lambda )=0,{\nabla }_{\lambda }\mathcal{L}(x,\lambda )=0⟺{\nabla }_{x}f(x)=\lambda {\nabla }_{x}g(x),g(x)=0$ The solution of the constrained optimization problem is always a Saddle Point of the Lagrangian function $\mathcal{L}(x,\lambda )$.

Facts

The gradient of constraint ${\nabla }_{x}g(x)$ and the gradient of the function ${\nabla }_{x}f(x)$ should be in same or directions.

Proof ${\nabla }_{x}f(x)+\lambda {\nabla }_{x}g(x)=0⟹{\nabla }_{x}f(x)=-\lambda {\nabla }_{x}g(x)$

If ${\nabla }_{x}f(x)$ and ${\nabla }_{x}g(x)$ don’t have same or directions, then there is the direction decreasing $f(x)$ subject to $g(x)=0$

If ${\nabla }_{x}f(x)$ and ${\nabla }_{x}g(x)$ have the same or opposite direction, then there is no direction decreasing $f(x)$ subject to $g(x)=0$