Stationary Point

Stationary points are critical points of a function where the derivative exists. Such a point can be either a local extremum (maximum or minimum) or a point of inflection (changing sign). Stationary points are extremely useful in optimisation, to finding the maximum and minimum of a system.

The local maxima and minima create the set of extrema of a function.

Definition §

Stationary Point

If $f$ is differentiable over some domain $D$, then $a\in D$ is a stationary point of $f$ iff:

$$f^{\prime }(a)=0$$

Stationary points can be either:

• Local Maximum: $f^{\prime \prime }(a)<0$
• Local Minimum: $f^{\prime \prime }(a)>0$
• Point of Inflection: $f^{\prime \prime }(a)=0$

Maximums and minimums together are known as extrema

Stationary Point (Multivariate)

If $f(x_1,\dots ,x_n)$ is a differentiable multivariate function over some domain $D$, then $(a_1,\dots ,a_n)\in D$ is a stationary point of $f$ iff:

$$\nabla f(a_1,\dots ,a_n)=\vec{0}$$

The nature of (bivariate) stationary points can be given by the second partial derivative test

Relation to Taylor Polynomials §

Let $f(x,y)$ be a bivariate function. Then, let $(a,b)$ be a stationary point for $f$. Notice how the first-order Taylor polynomial reduces to:

$$T_1(x,y)=f(a,b)+\cancel{\nabla f(a,b)}\vec{0}\cdot (x-a,y-b)=f(a,b)$$

That is, the Taylor approximation is terrible, because it is a constant value ($f(a,b)$) no matter the shape of $f$.

The second-order Taylor series returns:

$$\begin{array}{rl}{T}_2(x,y)& =T_1(x,y)+f_{xx}(a,b)\cdot \frac{(x-a{)}^2}{2!}+f_{xy}(a,b)\cdot \frac{(x-a)(y-b)}{1!1!}+f_{yy}(a,b)\cdot \frac{(y-b{)}^2}{2!}\\ & =f(a,b)+\frac{1}{2}[(x-a{)}^2\cdot f_{xx}(a,b)+2(x-a)(y-b)\cdot f_{xy}(a,b)+(y-b{)}^2\cdot f_{yy}(a,b)]\\ & =f(a,b)+\frac{1}{2}\left[\begin{array}{cc}x-a& y-b\end{array}\right]\left[\begin{array}{cc}{f}_{xx}(a,b)& f_{xy}(a,b)\\ f_{xy}(a,b)& f_{yy}(a,b)\end{array}\right]\left[\begin{array}{c}x-a\\ y-b\end{array}\right]\end{array}$$

Note: The final matrix is only possible if we assume $f\in C^2$ (it has second-order smoothness), due to the smoothness theorem for mixed partial derivatives

The matrix:

$$H_f=\left[\begin{array}{cc}{f}_{xx}(a,b)& f_{xy}(a,b)\\ f_{xy}(a,b)& f_{yy}(a,b)\end{array}\right]$$

is the Hessian Matrix