# ⚖️ Lagrange multipliers for constrained problems

By Better Wise

## The optimum moves onto the constraint surface

Unconstrained stationarity ($\nabla f = 0$) and the Hessian test from Module 2 assumed you could step in any direction. Once the feasible set is a surface such as $x + y = 4$ or $x^2 + y^2 = 2$, most directions are illegal, the gradient at the optimum is usually nonzero, and the Hessian test alone says nothing. Lagrange multipliers replace "gradient equals zero" with a condition that respects the surface, and they come with a bonus number, $\lambda$, that prices the constraint.

By the end of this lesson you can:

- write the Lagrangian for a problem with one or more equality constraints;
- solve the stationarity system by hand for the point and the multipliers;
- read $\lambda$ as the marginal change of the optimal value;
- confirm a hand solution with `scipy.optimize` and with a one-dimensional scan.

### Why the gradient turns parallel to the constraint

Take $\min f(x)$ subject to $g(x) = 0$, with $f$ smooth and $g$ scalar-valued. Let $x^\*$ be a local minimum and assume $\nabla g(x^\*) \neq 0$. Any smooth curve $x(t)$ that stays on the constraint surface with $x(0) = x^\*$ has velocity $d$ satisfying

$$
\nabla g(x^\*) \cdot d = 0,
$$

because $g(x(t))$ is identically zero, so its derivative vanishes. Feasible directions therefore form the tangent space, the orthogonal complement of $\nabla g$. Along such a curve the scalar function $t \mapsto f(x(t))$ has a local minimum at $t = 0$, so $\nabla f(x^\*) \cdot d = 0$ for every feasible $d$. A vector orthogonal to an entire $(n-1)$-dimensional subspace must be a multiple of the normal $\nabla g$:

$$
\nabla f(x^*) = \lambda \nabla g(x^*)
$$

for a single scalar $\lambda$. That is the whole geometry: **at a constrained optimum the objective gradient is parallel to the constraint gradient**, and $\lambda$ is the ratio between them.

Packaging this with the constraint gives the **Lagrangian**

$$
L(x, \lambda) = f(x) - \lambda\, g(x),
$$

and the stationarity system is $\nabla_x L = \nabla f - \lambda \nabla g = 0$ together with $\partial L/\partial \lambda = -g(x) = 0$. In $n$ variables with one equality constraint that is $n+1$ equations in $n+1$ unknowns. Differentiating with respect to $\lambda$ simply returns the constraint, which is why the method never loses it.

> **Note:** 
> The argument needs $\nabla g(x^\*) \neq 0$ (constraint qualification). If the constraint gradient vanishes at the optimum, no multiplier need exist, and solvers such as SLSQP also degrade there.

### Worked example 1: a linear constraint

Minimize $f(x,y) = x^2 + y^2$ subject to $x + y = 4$.

$$
L = x^2 + y^2 - \lambda(x + y - 4).
$$

Stationarity: $\partial_x L = 2x - \lambda = 0$, $\partial_y L = 2y - \lambda = 0$, $\partial_\lambda L = -(x + y - 4) = 0$. The first two give $x = y = \lambda/2$; the constraint then gives $\lambda = 4$, so $x^\* = y^\* = 2$ and $f^\* = 8$. Check the vector condition: $\nabla f = (4,4)$ and $\nabla g = (1,1)$, so $\nabla f = 4\nabla g$. Confirmed by hand, with $\lambda = 4$.

The same recipe scales. For $\min x^2 + y^2 + z^2$ subject to $x + 2y + 3z = 14$ with $a = (1,2,3)$, stationarity gives $2x = \lambda a$, so $x = (\lambda/2)a$. Substituting into the constraint: $(\lambda/2)\lVert a \rVert^2 = 7\lambda = 14$, hence $\lambda = 2$ and $x^\* = (1,2,3)$, $f^\* = 14$.

### Worked example 2: a curved constraint with four candidates

Optimize $f(x,y) = xy$ subject to $x^2 + y^2 = 2$. Here

$$
L = xy - \lambda(x^2 + y^2 - 2),
$$

so $y - 2\lambda x = 0$, $x - 2\lambda y = 0$, and $x^2 + y^2 = 2$. Substitution is messy; multiply the first equation by $y$ and the second by $x$ to get $y^2 = 2\lambda xy = x^2$. At a solution $x$ and $y$ are both nonzero (if one is zero, both stationarity equations force the other to be zero, violating the constraint), so $x^2 = y^2$. Combined with the constraint, the candidates are $(1,1), (1,-1), (-1,1), (-1,-1)$.

Evaluating: $(1,1)$ and $(-1,-1)$ give $f = 1$ with $\lambda = 1/2$; $(1,-1)$ and $(-1,1)$ give $f = -1$ with $\lambda = -1/2$. On a circle the maximum and minimum both sit at stationary points, so stationarity alone did not classify them: comparing values did. Note also that $\lambda$ changes sign between the max and the min.

### Reading $\lambda$ as a shadow price

Write the constraint as $g(x) = b$ and let $f^\*(b)$ be the optimal value. Differentiating $L = f - \lambda(g - b)$ in $b$ at the optimum gives $df^\*/db = \lambda$. In example 1, $f^\*(b) = b^2/2$, so $df^\*/db = b = 4$, matching $\lambda = 4$: relaxing the constraint by $\delta$ raises the optimal value by about $4\delta$. In example 2, with constraint $x^2 + y^2 = r^2$, the maximum is $r^2/2$, and its derivative with respect to the level $r^2$ is $1/2 = \lambda$. A zero multiplier means the constraint is not biting to first order.

### Confirming the hand solution numerically

Checking gradients against finite differences is the habit from earlier modules; here you also verify that $\nabla f - \lambda \nabla g$ vanishes and that the solver reaches the same point.

```python
import numpy as np
from scipy.optimize import minimize

# Example 1: min x^2 + y^2 s.t. x + y = 4
f = lambda v: v[0]**2 + v[1]**2
con = {"type": "eq", "fun": lambda v: v[0] + v[1] - 4}
res = minimize(f, x0=[0.0, 0.0], constraints=[con], method="SLSQP")

lam = 4.0
grad_f = np.array([2*res.x[0], 2*res.x[1]])
grad_g = np.array([1.0, 1.0])
print(res.x, res.fun)            # [2. 2.] 8.0
print(grad_f - lam*grad_g)       # ~ [0. 0.]

# Example 2: xy on the circle x^2 + y^2 = 2, parametrized scan
t = np.linspace(0.0, 2.0*np.pi, 400001)
vals = (np.sqrt(2)*np.cos(t)) * (np.sqrt(2)*np.sin(t))
print(vals.max(), vals.min())    # 1.0 -1.0
```

The scan is the honest check for example 2: it rules out any better feasible point that the stationarity system might have missed, including points where the multipliers fail.

### Algebra habits that prevent sign errors

- Fix one convention, $L = f - \lambda g$, and reuse it everywhere. Under that convention $\lambda = df^\*/db$; the opposite sign in $L$ flips every interpretation.
- Count equations before solving: one equality constraint in $n$ variables means $n+1$ equations, so there is no freedom left over.
- When the system is nonlinear, prefer symmetry moves (multiply by a variable, subtract equations) over blind substitution.
- Always test $\nabla g(x^\*) \neq 0$ at your answer, and never skip comparing candidate values: stationarity is necessary, not sufficient.
- Sanity-check $\lambda$ against a closed form or a finite-difference perturbation of the constraint level.

### What to carry into descent methods

- Constraints replace $\nabla f = 0$ with $\nabla f = \lambda \nabla g$: the gradient is absorbed by the surface normal.
- Solving means solving the full system, including $\partial L/\partial \lambda = 0$, which is just the constraint restated.
- $\lambda$ is a derivative, not decoration: it measures how much the optimal value moves when the constraint moves.

Next, gradient descent and learning rate behavior, where the Hessian from Module 2 tells you why a step size that works on one direction stalls or diverges on another.

---

## Contents

Part of [🧮 Mathematics for Machine Learning 2: Calculus and Optimization for Training Models](https://everwise.study/courses/mathematics-for-machine-learning-2-calculus-and-optimization-for-training.md)

- **📐 First-order derivatives: gradients, Jacobians and the chain rule**
  - [📐 From partial derivatives to the gradient vector](https://everwise.study/courses/mathematics-for-machine-learning-2-calculus-and-optimization-for-training/partial-derivatives-to-gradient-vector.md)
  - [🧮 Jacobians of vector-valued functions](https://everwise.study/courses/mathematics-for-machine-learning-2-calculus-and-optimization-for-training/jacobian-vector-valued-functions.md) (sign-in required)
  - [🔗 Chain rule for composed functions](https://everwise.study/courses/mathematics-for-machine-learning-2-calculus-and-optimization-for-training/chain-rule-composed-functions.md)
  - [✅ Checkpoint: first-order derivatives](https://everwise.study/courses/mathematics-for-machine-learning-2-calculus-and-optimization-for-training/checkpoint-first-order-derivatives.md)
- **📉 Second-order behavior and convexity**
  - [🌀 Hessians and second-order behavior](https://everwise.study/courses/mathematics-for-machine-learning-2-calculus-and-optimization-for-training/hessians-second-order-behavior.md)
  - [📈 Taylor expansions, convexity and local minima](https://everwise.study/courses/mathematics-for-machine-learning-2-calculus-and-optimization-for-training/taylor-expansions-convexity-local-minima.md)
  - [✅ Checkpoint: curvature and convexity](https://everwise.study/courses/mathematics-for-machine-learning-2-calculus-and-optimization-for-training/checkpoint-curvature-and-convexity.md)
- **⛰️ Constrained optimization and gradient descent**
  - [⚖️ Lagrange multipliers for constrained problems](https://everwise.study/courses/mathematics-for-machine-learning-2-calculus-and-optimization-for-training/lagrange-multipliers-constrained-problems.md) — this lesson
  - [🔽 Gradient descent and learning rate behavior](https://everwise.study/courses/mathematics-for-machine-learning-2-calculus-and-optimization-for-training/gradient-descent-learning-rate-behavior.md) (premium content)
  - [✅ Checkpoint: constrained optimization and descent](https://everwise.study/courses/mathematics-for-machine-learning-2-calculus-and-optimization-for-training/checkpoint-constrained-optimization-descent.md) (premium content)
- [Course cheatsheet](https://everwise.study/courses/mathematics-for-machine-learning-2-calculus-and-optimization-for-training/cheatsheet.md) (premium content)