# 📈 Taylor expansions, convexity and local minima By Better Wise The previous lesson gave you the Hessian $H(x)$, the symmetric matrix of second partial derivatives, and a local second-order test at a critical point. This lesson turns that local picture into a global one: we build the Taylor model that $H$ defines, read curvature off it, and use convexity to decide whether a local minimum is also the global minimum. **By the end of this lesson you can:** - write the first- and second-order Taylor models of $f$ around a base point $x$ - reduce a multivariate expansion to a scalar one along $\phi(t) = f(x + tp)$ - state the Hessian test for convexity and say exactly where it must hold - classify a critical point as minimum, saddle, or inconclusive from $H(x^*)$ - explain why convexity upgrades a local minimum to a global one ## Building the Taylor model one dimension at a time The one technique worth memorizing: never expand in $n$ variables at once. Fix a base point $x$ and a step $p$, then define the scalar function $\phi(t) = f(x + tp)$. The chain rule gives $$ \phi'(t) = \nabla f(x + tp)^T p, \qquad \phi''(t) = p^T H(x + tp)\, p. $$ Now Taylor in the single variable $t$ around $0$, and set $t = 1$: $$ f(x+p) = f(x) + \nabla f(x)^T p + \frac{1}{2} p^T H(x)\, p + O(\|p\|^3) $$ The first-order truncation is the tangent hyperplane. The second-order truncation is a quadratic bowl whose shape is exactly $H(x)$. **Worked example 1.** Take $f(x,y) = x^2 y + y^2 + x$ at the base point $(1,1)$, with step $p = (p, q)$. Then $f(1,1) = 3$, $\nabla f = (2xy + 1,\; x^2 + 2y) = (3,3)$, and $H = \begin{pmatrix} 2y & 2x \\ 2x & 2 \end{pmatrix} = \begin{pmatrix} 2 & 2 \\ 2 & 2 \end{pmatrix}$. So $$ f(1+p, 1+q) \approx 3 + 3p + 3q + \tfrac{1}{2}\left(2p^2 + 4pq + 2q^2\right) = 3 + 3p + 3q + (p+q)^2. $$ Note the Hessian is singular here: its eigenvalues are $0$ and $4$. The quadratic term vanishes along the direction $(1,-1)$, which is a warning that the model says nothing in that direction. ```python import numpy as np def f(v): x, y = v; return x**2 * y + y**2 + x def grad(v): x, y = v; return np.array([2*x*y + 1, x**2 + 2*y]) def hess(v): x, y = v; return np.array([[2*y, 2*x], [2*x, 2.0]]) x0, p = np.array([1.0, 1.0]), np.array([0.01, 0.01]) first = f(x0) + grad(x0) @ p second = first + 0.5 * p @ hess(x0) @ p print(f(x0 + p), first, second, np.linalg.eigvalsh(hess(x0))) # 3.060401 3.06 3.0604 [0. 4.] ``` The second-order model matches the true value to four decimals while the linear model is off by $4\times10^{-4}$. ## Reading the verdict at a critical point At a critical point $\nabla f(x^*) = 0$, the expansion collapses to $f(x^*+p) - f(x^*) \approx \tfrac12 p^T H p$. Diagonalize $H = \sum_i \lambda_i q_i q_i^T$, so $$ \tfrac12 p^T H p = \tfrac12 \sum_i \lambda_i (q_i^T p)^2. $$ Each eigenvalue contributes with its own sign. That single formula produces the whole classification. ```mermaid flowchart TD A["Critical point: gradient is zero"] --> B["Eigenvalues of the Hessian"] B --> C["All strictly positive: strict local minimum"] B --> D["At least one negative: not a minimum"] B --> E["All nonnegative, some zero: inconclusive"] E --> F["Inspect higher-order terms along null directions"] ``` Two functions show why the zero-eigenvalue branch is genuinely undecided. Both $f_1(x,y) = x^2 + y^4$ and $f_2(x,y) = x^2 + y^3$ have $\nabla f(0,0) = 0$ and $H(0,0) = \mathrm{diag}(2,0)$, whose eigenvalues are $2$ and $0$. Yet $f_1$ has a strict minimum at the origin, while $f_2$ decreases along $y < 0$ and has no minimum there. The Hessian alone cannot separate them. ## Convexity: the same test, demanded everywhere A function is convex on a convex domain if for all $x, y$ and $\theta \in [0,1]$, $$ f(\theta x + (1-\theta) y) \le \theta f(x) + (1-\theta) f(y). $$ For twice continuously differentiable $f$ on a convex domain, this is equivalent to $H(x) \succeq 0$ at **every** point of that domain. Requiring $H(x) \succ 0$ everywhere gives strict convexity. **Worked example 2.** For $f(x,y) = x^2 + xy + y^2$, $H = \begin{pmatrix} 2 & 1 \\ 1 & 2 \end{pmatrix}$ with eigenvalues $1$ and $3$, both positive at every point, so $f$ is strictly convex. Solving $\nabla f = (2x+y,\; x+2y) = 0$ gives $(0,0)$, and by convexity that critical point is the global minimum. **Contrast.** For $f(x,y) = x^2 - y^2$, $H = \mathrm{diag}(2,-2)$ is indefinite, the origin is a saddle, and no minimum exists. For $f(x,y) = x^2$, $H = \mathrm{diag}(2,0) \succeq 0$ everywhere: convex but not strictly, and the minimizers form the whole line $x = 0$. ## From local to global Convexity is what closes the gap. For convex differentiable $f$ the first-order inequality $f(y) \ge f(x) + \nabla f(x)^T (y - x)$ holds for all $x, y$. Set $x = x^*$ with $\nabla f(x^*) = 0$ and you get $f(y) \ge f(x^*)$ for every $y$: a critical point is automatically global. **Worked example 3.** Let $f(x) = \tfrac12 x^T A x - b^T x$ with $A$ symmetric and $A \succ 0$. Then $\nabla f = Ax - b$ and $H = A$, constant. Since $A \succ 0$, $f$ is strictly convex and $x^* = A^{-1}b$ is the unique global minimizer. **Where it fails.** For $f(x) = x^3 - 3x$ we find $f'(x) = 3x^2 - 3 = 0$ at $x = \pm 1$, and $f''(1) = 6 > 0$ confirms a local minimum with $f(1) = -2$. But $f(x) \to -\infty$ as $x \to -\infty$, so that local minimum is not global. The Hessian is indefinite somewhere, and that is exactly the loophole. The second-order Taylor model adds the term _____ to the linear model. A twice differentiable function is convex on a convex domain exactly when its Hessian is _____ at every point of that domain. If the Hessian is positive definite at a critical point, that point is a _____. If the Hessian at a critical point has a negative eigenvalue, the point is _____. For a convex differentiable function, every critical point is a _____. ## Practical rules for Taylor models and Hessian tests - Keep the factor $\tfrac12$: the quadratic term is $\tfrac12 p^T H p$, not $p^T H p$. Forgetting it is the most common error, and the numeric check above catches it in one line. - Expand around a base point and a separate step $p$. Mixing up $x$ and $p$ inside the Hessian ruins the model. - A Hessian test at a single point tells you about a neighborhood, never about the whole function. Global convexity needs $H \succeq 0$ on a convex domain, checked symbolically or on a grid. - Use `np.linalg.eigvalsh` rather than `eig`: $H$ is symmetric, so you get real eigenvalues, sorted, with no complex round-off noise. - Singular Hessian at a critical point is not a failure, it is an instruction: expand to higher order along the null directions, as $x^2 + y^3$ and $x^2 + y^4$ demand. - The remainder is $O(\|p\|^3)$, so the quadratic model is only trustworthy for small steps. If a check disagrees at $p = 0.1$, retry at $p = 0.01$ before doubting the derivative. ## What to carry into the checkpoint Taylor expansion turns $H$ into a usable model: linear term for direction, quadratic term for curvature. Eigenvalue signs then classify a critical point, with the semidefinite-singular case left open. Convexity, meaning $H \succeq 0$ everywhere on a convex domain, is what converts a local verdict into a global one. Next up is the checkpoint on curvature and convexity, where you will read Hessians, classify critical points, and justify convexity claims end to end. ## Review deck Front: What is the second-order Taylor model of f at x with step p? Back: f(x+p) ≈ f(x) + gradient(x)^T p + one half p^T H(x) p. Front: What is the first-order Taylor model geometrically? Back: The tangent hyperplane f(x) + gradient(x)^T p. Front: How do you derive a multivariate Taylor expansion? Back: Set phi(t) = f(x + t p) and expand the scalar function in t. Front: What is phi''(t) for phi(t) = f(x + t p)? Back: p^T H(x + t p) p. Front: Hessian test for convexity on a convex domain? Back: H(x) positive semidefinite at every point of that domain. Front: Sufficient condition for strict convexity? Back: H(x) positive definite everywhere on the domain. Front: Sufficient second-order condition for a strict local minimum? Back: Gradient zero and Hessian positive definite at that point. Front: What does a negative eigenvalue of H(x*) imply at a critical point? Back: x* is not a local minimum; it is a saddle or a maximum. Front: What if H(x*) is positive semidefinite but singular at a critical point? Back: The test is inconclusive; higher-order terms along null directions decide. Front: Why are x^2 + y^3 and x^2 + y^4 a useful pair? Back: Same Hessian diag(2,0) at the origin, different verdicts: minimum versus no minimum. Front: For convex differentiable f, what does gradient zero imply? Back: A global minimum, by the first-order convexity inequality. Front: Is H positive semidefinite at one point enough for convexity? Back: No; it must hold at every point of a convex domain. Front: Hessian of f(x) = one half x^T A x - b^T x? Back: H = A, so A positive definite gives a unique global minimizer at A inverse b. Front: How large a step p keeps the Taylor model accurate? Back: The remainder is O(norm(p)^3), so only small steps are trustworthy. Front: Why use eigvalsh instead of eig on a Hessian? Back: H is symmetric, so eigenvalues are real and returned sorted without complex noise. --- ## Contents Part of [🧮 Mathematics for Machine Learning 2: Calculus and Optimization for Training Models](https://everwise.study/courses/mathematics-for-machine-learning-2-calculus-and-optimization-for-training.md) - **📐 First-order derivatives: gradients, Jacobians and the chain rule** - [📐 From partial derivatives to the gradient vector](https://everwise.study/courses/mathematics-for-machine-learning-2-calculus-and-optimization-for-training/partial-derivatives-to-gradient-vector.md) - [🧮 Jacobians of vector-valued functions](https://everwise.study/courses/mathematics-for-machine-learning-2-calculus-and-optimization-for-training/jacobian-vector-valued-functions.md) (sign-in required) - [🔗 Chain rule for composed functions](https://everwise.study/courses/mathematics-for-machine-learning-2-calculus-and-optimization-for-training/chain-rule-composed-functions.md) - [✅ Checkpoint: first-order derivatives](https://everwise.study/courses/mathematics-for-machine-learning-2-calculus-and-optimization-for-training/checkpoint-first-order-derivatives.md) - **📉 Second-order behavior and convexity** - [🌀 Hessians and second-order behavior](https://everwise.study/courses/mathematics-for-machine-learning-2-calculus-and-optimization-for-training/hessians-second-order-behavior.md) - [📈 Taylor expansions, convexity and local minima](https://everwise.study/courses/mathematics-for-machine-learning-2-calculus-and-optimization-for-training/taylor-expansions-convexity-local-minima.md) — this lesson - [✅ Checkpoint: curvature and convexity](https://everwise.study/courses/mathematics-for-machine-learning-2-calculus-and-optimization-for-training/checkpoint-curvature-and-convexity.md) - **⛰️ Constrained optimization and gradient descent** - [⚖️ Lagrange multipliers for constrained problems](https://everwise.study/courses/mathematics-for-machine-learning-2-calculus-and-optimization-for-training/lagrange-multipliers-constrained-problems.md) - [🔽 Gradient descent and learning rate behavior](https://everwise.study/courses/mathematics-for-machine-learning-2-calculus-and-optimization-for-training/gradient-descent-learning-rate-behavior.md) (premium content) - [✅ Checkpoint: constrained optimization and descent](https://everwise.study/courses/mathematics-for-machine-learning-2-calculus-and-optimization-for-training/checkpoint-constrained-optimization-descent.md) (premium content) - [Course cheatsheet](https://everwise.study/courses/mathematics-for-machine-learning-2-calculus-and-optimization-for-training/cheatsheet.md) (premium content)