Hessians and second-order behavior

By Better Wise

Why One Derivative Is Not Enough

In module 1 we built the gradient $\nabla f(x)$, the direction of steepest ascent, and the Jacobian, which is just that same derivative stacked for vector-valued maps. Recall the key line: the gradient gives the slope at a point. It says nothing about how that slope changes when you take a step. That missing information is curvature, and curvature is exactly what separates a critical point that is a minimum from one that is a saddle.

After this lesson you can:

• assemble a Hessian by hand from a scalar function
• confirm it with finite differences in Python
• read its eigenvalues as curvature along principal directions
• classify a critical point as minimum, maximum or saddle
• recognise when the second-order test tells you nothing

Building the Hessian Matrix

For $f:\mathbb{R}^n \to \mathbb{R}$, twice differentiable, the Hessian at $x$ is the $n \times n$ matrix

$$ H(x)_{ij} = \frac{\partial^2 f}{\partial x_i , \partial x_j} $$

The structural fact worth memorising: the Hessian is the _____, so entry $(i,j)$ is _____. Everything you already know about Jacobians transfers directly.

If the second partials are continuous, mixed partials commute, so $H$ is symmetric. A real symmetric matrix has real eigenvalues and an orthonormal eigenbasis, which is precisely why eigenvalues become the natural language for curvature.

The bridge to curvature is the second directional derivative. For a unit vector $v$,

$$ \left.\frac{d^2}{dt^2} f(x + tv)\right|_{t=0} = v^\top H(x) , v $$

So the second directional derivative along any unit direction is _____. The Hessian is a quadratic form, not just a table of numbers.

Three Worked Examples, Each Checked in Code

Example 1 (a saddle). Take $f(x,y) = x^2 + 3xy + 2y^2$. Then $\nabla f = (2x + 3y,; 3x + 4y)$, and

$$ H = \begin{pmatrix} 2 & 3 \ 3 & 4 \end{pmatrix} $$

The only critical point is the origin. $\det H = 8 - 9 = -1 < 0$, so $H$ is indefinite. The eigenvalues are $3 \pm \sqrt{10} \approx 6.162$ and $-0.162$: the function rises along one direction and falls along another. The origin is a saddle.

import numpy as np

def f(x):
    return x[0]**2 + 3*x[0]*x[1] + 2*x[1]**2

def num_hess(f, x, h=1e-4):
    n = x.size
    H = np.zeros((n, n))
    for i in range(n):
        for j in range(n):
            ei, ej = np.eye(n)[i] * h, np.eye(n)[j] * h
            H[i, j] = (f(x + ei + ej) - f(x + ei - ej)
                       - f(x - ei + ej) + f(x - ei - ej)) / (4 * h * h)
    return H

H = num_hess(f, np.array([0.0, 0.0]))
print(H)                        # [[2. 3.] [3. 4.]]
print(np.linalg.eigvalsh(H))    # [ 6.162, -0.162] -> indefinite -> saddle

Example 2 (a minimum with anisotropic curvature). Take $f(x,y) = x^2 + 4xy + 5y^2$. Then $H = \begin{pmatrix} 2 & 4 \ 4 & 10 \end{pmatrix}$, with $\det H = 4 > 0$ and $\operatorname{tr} H = 12 > 0$, so $H$ is positive definite. Eigenvalues are $6 \pm 4\sqrt{2} \approx 11.657$ and $0.343$. The condition number is about 34: the bowl is 34 times steeper along one axis than the other. That number will matter a lot when we reach gradient descent.

Example 3 (a degenerate point). Take $f(x,y) = x^2 + y^4$. Then $H = \begin{pmatrix} 2 & 0 \ 0 & 12y^2 \end{pmatrix}$, which at the origin is $\begin{pmatrix} 2 & 0 \ 0 & 0 \end{pmatrix}$. That is positive semidefinite, so the test is inconclusive. But $f(x,y) \ge 0$ everywhere and equals zero only at the origin, so the origin is a global minimum. The second-order test is a sufficient condition, not a definition.

Eigenvalues Read Out the Curvature

• Each eigenvector $u_i$ of $H$ is a principal direction; the curvature along it is its eigenvalue $\lambda_i$.
• $\lambda > 0$ means the surface curves upward (bowl-like), $\lambda < 0$ curves downward (dome-like), $\lambda = 0$ is flat to second order.
• If $\lambda_{\min} \approx 0$, the landscape is nearly flat in that direction: optimisation will crawl.
• Sanity checks: $\operatorname{tr} H = \sum_i \lambda_i = \nabla^2 f$ (the Laplacian) and $\det H = \prod_i \lambda_i$. Compute both by hand; they catch most algebra slips.

Classifying Critical Points

Assume first that $\nabla f(x^) = 0$. Then only the eigenvalues of $H(x^)$ decide the type.

In two dimensions there is a shortcut using the determinant: $\det H > 0$ with $H_{11} > 0$ gives a minimum, $\det H > 0$ with $H_{11} < 0$ gives a maximum, $\det H < 0$ gives a saddle, and $\det H = 0$ gives no answer. Note the missing case: $\det > 0$ alone is not enough, you also need the sign of a diagonal entry.

Warning: The second-order test only classifies critical points. Evaluating the Hessian at a point where the gradient is nonzero and calling it a minimum is a category error. Check $\nabla f = 0$ first, every time.

At a critical point, a Hessian with both positive and negative eigenvalues indicates a _____. A Hessian with one _____ and all others of one sign leaves the second-order test inconclusive. The trace of the Hessian equals the _____, which is also the Laplacian of f.

Reading a Hessian Without Fooling Yourself

• Differentiate to get $H$ by hand, then run num_hess immediately. A mismatched entry usually means one dropped cross term.
• Use $h \approx 10^{-4}$ for the finite-difference Hessian. Smaller $h$ amplifies floating-point cancellation, larger $h$ picks up third-order error.
• Never use np.linalg.eig for a symmetric matrix; np.linalg.eigvalsh is faster and returns sorted real values.
• The classic error: reading $\det H > 0$ as "minimum" and skipping the sign of the trace or of $H_{11}$.
• Another classic: computing $H$ at the origin out of habit when $H$ depends on $x$, as in $x^2 + y^4$.
• Treat a small $\lambda_{\min}$ as a warning label, not a technicality. Ill-conditioned curvature is the usual reason descent stalls.

What to Take Away

The Hessian is the Jacobian of the gradient, a symmetric matrix of second partials whose quadratic form $v^\top H v$ gives the second directional derivative along $v$. At a critical point, all-positive eigenvalues mean a minimum, all-negative a maximum, mixed signs a saddle, and any zero eigenvalue means the test is silent. Hand-derive it, sanity-check trace and determinant, then confirm with eigvalsh and a finite-difference Hessian.

Next lesson we lift the Hessian out of single points: the Taylor expansion built from $H$ gives a local quadratic model, and that model is the bridge from curvature to convexity and to genuinely global minima.

Front: Define the Hessian of f: R^n -> R. Back: The n x n matrix of second partials, H_ij = d2f / dx_i dx_j.

Front: How does the Hessian relate to the gradient? Back: It is the Jacobian of the gradient field, H = J(grad f).

Front: When is the Hessian guaranteed to be symmetric? Back: When all second partials are continuous, so mixed partials commute.

Front: What is the second directional derivative of f along a unit vector v? Back: v^T H(x) v, the quadratic form of the Hessian.

Front: What is the curvature of f along an eigenvector u of H? Back: Its eigenvalue lambda.

Front: At a critical point, what does an all-positive eigenvalue spectrum mean? Back: A strict local minimum.

Front: At a critical point, what does a mixed-sign eigenvalue spectrum mean? Back: A saddle point.

Front: The Hessian at a critical point is the zero matrix. What now? Back: The second-order test is inconclusive; higher-order or global reasoning is needed.

Front: What do trace(H) and det(H) give you? Back: The sum and the product of the eigenvalues, useful as hand-computation sanity checks.

Front: State the 2D determinant shortcut for classifying a critical point. Back: det > 0 and H11 > 0: min; det > 0 and H11 < 0: max; det < 0: saddle; det = 0: inconclusive.

Front: What does a large condition number lambda_max / lambda_min say about the landscape? Back: Strongly anisotropic curvature: very steep in one direction, nearly flat in another.

Front: What is the Hessian of f(x,y) = x^2 + 3xy + 2y^2, and what point is the origin? Back: [[2,3],[3,4]], det = -1, indefinite, so the origin is a saddle.

Front: Why check a hand-derived Hessian with finite differences? Back: Because cross-term mistakes are easy to make and a numeric check exposes them immediately.

Pitfalls when reading curvature by hand

Check these before you trust a classification.

• Differentiate twice, not once. The Hessian entry $H_{ij}$ is $\partial_i(\partial_j f)$. For $f(x,y) = x^2 y$, you get $H_{xy} = 2x$, not $y$. Write the gradient first, then differentiate each component again.
• Symmetry is a free consistency check. If your $H_{12} \neq H_{21}$, you made an algebra slip, because mixed partials commute on smooth functions.
• Classify only at critical points. Eigenvalues of $H(x)$ say nothing about min/max unless $\nabla f(x) = 0$. Off a critical point, negative curvature just means "the slope is decreasing here".
• Zero eigenvalues are a warning, not an answer. A semidefinite Hessian ($\lambda = 0$ present) makes the second-order test inconclusive. Fall back to the function itself or higher-order terms.
• Trace and determinant are your hand calculator. $\mathrm{tr}(H) = \sum \lambda_i$ and $\det(H) = \prod \lambda_i$. Compute both, and compare with what your eigenvalues claim.
• Always run the numeric check. One line of central differences on the gradient costs seconds and catches the $-1$ on a cross term that a sign error loves to hide.

The habit worth building: derive by hand, verify numerically, then interpret. Never interpret first.

What to carry forward

• The Hessian is the Jacobian of the gradient: $H(x) \in \mathbb{R}^{n \times n}$, symmetric for twice-differentiable $f$.
• Eigenvalues are directional curvature. Positive means the function curves upward along that eigenvector, negative means downward, magnitude sets how sharply.
• At a critical point: all $\lambda > 0$ is a local minimum, all $\lambda < 0$ is a local maximum, mixed signs is a saddle, and a zero eigenvalue leaves the test inconclusive.
• In two dimensions, the determinant shortcut classifies in one line: $\det > 0$ with $H_{11} > 0$ is a min, $\det > 0$ with $H_{11} < 0$ is a max, $\det < 0$ is a saddle.
• The condition number $\lambda_{\max}/\lambda_{\min}$ measures how anisotropic the bowl is, which is exactly what will control gradient descent later.

Next lesson turns this local picture into a global one: the second-order Taylor expansion built from $H$, and how positive semidefiniteness everywhere upgrades "local minimum" to "global minimum".