📈 Taylor expansions, convexity and local minima

By Better Wise

The previous lesson gave you the Hessian $H(x)$, the symmetric matrix of second partial derivatives, and a local second-order test at a critical point. This lesson turns that local picture into a global one: we build the Taylor model that $H$ defines, read curvature off it, and use convexity to decide whether a local minimum is also the global minimum.

By the end of this lesson you can:

  • write the first- and second-order Taylor models of $f$ around a base point $x$
  • reduce a multivariate expansion to a scalar one along $\phi(t) = f(x + tp)$
  • state the Hessian test for convexity and say exactly where it must hold
  • classify a critical point as minimum, saddle, or inconclusive from $H(x^*)$
  • explain why convexity upgrades a local minimum to a global one

Building the Taylor model one dimension at a time

The one technique worth memorizing: never expand in $n$ variables at once. Fix a base point $x$ and a step $p$, then define the scalar function $\phi(t) = f(x + tp)$. The chain rule gives

$$ \phi'(t) = \nabla f(x + tp)^T p, \qquad \phi''(t) = p^T H(x + tp), p. $$

Now Taylor in the single variable $t$ around $0$, and set $t = 1$:

$$ f(x+p) = f(x) + \nabla f(x)^T p + \frac{1}{2} p^T H(x), p + O(|p|^3) $$

The first-order truncation is the tangent hyperplane. The second-order truncation is a quadratic bowl whose shape is exactly $H(x)$.

Worked example 1. Take $f(x,y) = x^2 y + y^2 + x$ at the base point $(1,1)$, with step $p = (p, q)$. Then $f(1,1) = 3$, $\nabla f = (2xy + 1,; x^2 + 2y) = (3,3)$, and $H = \begin{pmatrix} 2y & 2x \ 2x & 2 \end{pmatrix} = \begin{pmatrix} 2 & 2 \ 2 & 2 \end{pmatrix}$. So

$$ f(1+p, 1+q) \approx 3 + 3p + 3q + \tfrac{1}{2}\left(2p^2 + 4pq + 2q^2\right) = 3 + 3p + 3q + (p+q)^2. $$

Note the Hessian is singular here: its eigenvalues are $0$ and $4$. The quadratic term vanishes along the direction $(1,-1)$, which is a warning that the model says nothing in that direction.

import numpy as np

def f(v):    x, y = v; return x**2 * y + y**2 + x
def grad(v): x, y = v; return np.array([2*x*y + 1, x**2 + 2*y])
def hess(v): x, y = v; return np.array([[2*y, 2*x], [2*x, 2.0]])

x0, p = np.array([1.0, 1.0]), np.array([0.01, 0.01])
first  = f(x0) + grad(x0) @ p
second = first + 0.5 * p @ hess(x0) @ p
print(f(x0 + p), first, second, np.linalg.eigvalsh(hess(x0)))
# 3.060401  3.06  3.0604  [0. 4.]

The second-order model matches the true value to four decimals while the linear model is off by $4\times10^{-4}$.

Reading the verdict at a critical point

At a critical point $\nabla f(x^) = 0$, the expansion collapses to $f(x^+p) - f(x^*) \approx \tfrac12 p^T H p$. Diagonalize $H = \sum_i \lambda_i q_i q_i^T$, so

$$ \tfrac12 p^T H p = \tfrac12 \sum_i \lambda_i (q_i^T p)^2. $$

Each eigenvalue contributes with its own sign. That single formula produces the whole classification.

flowchart TD
    A["Critical point: gradient is zero"] --> B["Eigenvalues of the Hessian"]
    B --> C["All strictly positive: strict local minimum"]
    B --> D["At least one negative: not a minimum"]
    B --> E["All nonnegative, some zero: inconclusive"]
    E --> F["Inspect higher-order terms along null directions"]

Two functions show why the zero-eigenvalue branch is genuinely undecided. Both $f_1(x,y) = x^2 + y^4$ and $f_2(x,y) = x^2 + y^3$ have $\nabla f(0,0) = 0$ and $H(0,0) = \mathrm{diag}(2,0)$, whose eigenvalues are $2$ and $0$. Yet $f_1$ has a strict minimum at the origin, while $f_2$ decreases along $y < 0$ and has no minimum there. The Hessian alone cannot separate them.

Convexity: the same test, demanded everywhere

A function is convex on a convex domain if for all $x, y$ and $\theta \in [0,1]$,

$$ f(\theta x + (1-\theta) y) \le \theta f(x) + (1-\theta) f(y). $$

For twice continuously differentiable $f$ on a convex domain, this is equivalent to $H(x) \succeq 0$ at every point of that domain. Requiring $H(x) \succ 0$ everywhere gives strict convexity.

Worked example 2. For $f(x,y) = x^2 + xy + y^2$, $H = \begin{pmatrix} 2 & 1 \ 1 & 2 \end{pmatrix}$ with eigenvalues $1$ and $3$, both positive at every point, so $f$ is strictly convex. Solving $\nabla f = (2x+y,; x+2y) = 0$ gives $(0,0)$, and by convexity that critical point is the global minimum.

Contrast. For $f(x,y) = x^2 - y^2$, $H = \mathrm{diag}(2,-2)$ is indefinite, the origin is a saddle, and no minimum exists. For $f(x,y) = x^2$, $H = \mathrm{diag}(2,0) \succeq 0$ everywhere: convex but not strictly, and the minimizers form the whole line $x = 0$.

From local to global

Convexity is what closes the gap. For convex differentiable $f$ the first-order inequality $f(y) \ge f(x) + \nabla f(x)^T (y - x)$ holds for all $x, y$. Set $x = x^$ with $\nabla f(x^) = 0$ and you get $f(y) \ge f(x^*)$ for every $y$: a critical point is automatically global.

Worked example 3. Let $f(x) = \tfrac12 x^T A x - b^T x$ with $A$ symmetric and $A \succ 0$. Then $\nabla f = Ax - b$ and $H = A$, constant. Since $A \succ 0$, $f$ is strictly convex and $x^* = A^{-1}b$ is the unique global minimizer.

Where it fails. For $f(x) = x^3 - 3x$ we find $f'(x) = 3x^2 - 3 = 0$ at $x = \pm 1$, and $f''(1) = 6 > 0$ confirms a local minimum with $f(1) = -2$. But $f(x) \to -\infty$ as $x \to -\infty$, so that local minimum is not global. The Hessian is indefinite somewhere, and that is exactly the loophole.

The second-order Taylor model adds the term _____ to the linear model. A twice differentiable function is convex on a convex domain exactly when its Hessian is _____ at every point of that domain. If the Hessian is positive definite at a critical point, that point is a _____. If the Hessian at a critical point has a negative eigenvalue, the point is _____. For a convex differentiable function, every critical point is a _____.

Practical rules for Taylor models and Hessian tests

  • Keep the factor $\tfrac12$: the quadratic term is $\tfrac12 p^T H p$, not $p^T H p$. Forgetting it is the most common error, and the numeric check above catches it in one line.
  • Expand around a base point and a separate step $p$. Mixing up $x$ and $p$ inside the Hessian ruins the model.
  • A Hessian test at a single point tells you about a neighborhood, never about the whole function. Global convexity needs $H \succeq 0$ on a convex domain, checked symbolically or on a grid.
  • Use np.linalg.eigvalsh rather than eig: $H$ is symmetric, so you get real eigenvalues, sorted, with no complex round-off noise.
  • Singular Hessian at a critical point is not a failure, it is an instruction: expand to higher order along the null directions, as $x^2 + y^3$ and $x^2 + y^4$ demand.
  • The remainder is $O(|p|^3)$, so the quadratic model is only trustworthy for small steps. If a check disagrees at $p = 0.1$, retry at $p = 0.01$ before doubting the derivative.

What to carry into the checkpoint

Taylor expansion turns $H$ into a usable model: linear term for direction, quadratic term for curvature. Eigenvalue signs then classify a critical point, with the semidefinite-singular case left open. Convexity, meaning $H \succeq 0$ everywhere on a convex domain, is what converts a local verdict into a global one.

Next up is the checkpoint on curvature and convexity, where you will read Hessians, classify critical points, and justify convexity claims end to end.

Review deck

Front: What is the second-order Taylor model of f at x with step p? Back: f(x+p) ≈ f(x) + gradient(x)^T p + one half p^T H(x) p.

Front: What is the first-order Taylor model geometrically? Back: The tangent hyperplane f(x) + gradient(x)^T p.

Front: How do you derive a multivariate Taylor expansion? Back: Set phi(t) = f(x + t p) and expand the scalar function in t.

Front: What is phi''(t) for phi(t) = f(x + t p)? Back: p^T H(x + t p) p.

Front: Hessian test for convexity on a convex domain? Back: H(x) positive semidefinite at every point of that domain.

Front: Sufficient condition for strict convexity? Back: H(x) positive definite everywhere on the domain.

Front: Sufficient second-order condition for a strict local minimum? Back: Gradient zero and Hessian positive definite at that point.

Front: What does a negative eigenvalue of H(x*) imply at a critical point? Back: x* is not a local minimum; it is a saddle or a maximum.

Front: What if H(x*) is positive semidefinite but singular at a critical point? Back: The test is inconclusive; higher-order terms along null directions decide.

Front: Why are x^2 + y^3 and x^2 + y^4 a useful pair? Back: Same Hessian diag(2,0) at the origin, different verdicts: minimum versus no minimum.

Front: For convex differentiable f, what does gradient zero imply? Back: A global minimum, by the first-order convexity inequality.

Front: Is H positive semidefinite at one point enough for convexity? Back: No; it must hold at every point of a convex domain.

Front: Hessian of f(x) = one half x^T A x - b^T x? Back: H = A, so A positive definite gives a unique global minimizer at A inverse b.

Front: How large a step p keeps the Taylor model accurate? Back: The remainder is O(norm(p)^3), so only small steps are trustworthy.

Front: Why use eigvalsh instead of eig on a Hessian? Back: H is symmetric, so eigenvalues are real and returned sorted without complex noise.