⚖️ Lagrange multipliers for constrained problems

By Better Wise

The optimum moves onto the constraint surface

Unconstrained stationarity ($\nabla f = 0$) and the Hessian test from Module 2 assumed you could step in any direction. Once the feasible set is a surface such as $x + y = 4$ or $x^2 + y^2 = 2$, most directions are illegal, the gradient at the optimum is usually nonzero, and the Hessian test alone says nothing. Lagrange multipliers replace "gradient equals zero" with a condition that respects the surface, and they come with a bonus number, $\lambda$, that prices the constraint.

By the end of this lesson you can:

  • write the Lagrangian for a problem with one or more equality constraints;
  • solve the stationarity system by hand for the point and the multipliers;
  • read $\lambda$ as the marginal change of the optimal value;
  • confirm a hand solution with scipy.optimize and with a one-dimensional scan.

Why the gradient turns parallel to the constraint

Take $\min f(x)$ subject to $g(x) = 0$, with $f$ smooth and $g$ scalar-valued. Let $x^*$ be a local minimum and assume $\nabla g(x^*) \neq 0$. Any smooth curve $x(t)$ that stays on the constraint surface with $x(0) = x^*$ has velocity $d$ satisfying

$$ \nabla g(x^*) \cdot d = 0, $$

because $g(x(t))$ is identically zero, so its derivative vanishes. Feasible directions therefore form the tangent space, the orthogonal complement of $\nabla g$. Along such a curve the scalar function $t \mapsto f(x(t))$ has a local minimum at $t = 0$, so $\nabla f(x^*) \cdot d = 0$ for every feasible $d$. A vector orthogonal to an entire $(n-1)$-dimensional subspace must be a multiple of the normal $\nabla g$:

$$ \nabla f(x^) = \lambda \nabla g(x^) $$

for a single scalar $\lambda$. That is the whole geometry: at a constrained optimum the objective gradient is parallel to the constraint gradient, and $\lambda$ is the ratio between them.

Packaging this with the constraint gives the Lagrangian

$$ L(x, \lambda) = f(x) - \lambda, g(x), $$

and the stationarity system is $\nabla_x L = \nabla f - \lambda \nabla g = 0$ together with $\partial L/\partial \lambda = -g(x) = 0$. In $n$ variables with one equality constraint that is $n+1$ equations in $n+1$ unknowns. Differentiating with respect to $\lambda$ simply returns the constraint, which is why the method never loses it.

Note: The argument needs $\nabla g(x^*) \neq 0$ (constraint qualification). If the constraint gradient vanishes at the optimum, no multiplier need exist, and solvers such as SLSQP also degrade there.

Worked example 1: a linear constraint

Minimize $f(x,y) = x^2 + y^2$ subject to $x + y = 4$.

$$ L = x^2 + y^2 - \lambda(x + y - 4). $$

Stationarity: $\partial_x L = 2x - \lambda = 0$, $\partial_y L = 2y - \lambda = 0$, $\partial_\lambda L = -(x + y - 4) = 0$. The first two give $x = y = \lambda/2$; the constraint then gives $\lambda = 4$, so $x^* = y^* = 2$ and $f^* = 8$. Check the vector condition: $\nabla f = (4,4)$ and $\nabla g = (1,1)$, so $\nabla f = 4\nabla g$. Confirmed by hand, with $\lambda = 4$.

The same recipe scales. For $\min x^2 + y^2 + z^2$ subject to $x + 2y + 3z = 14$ with $a = (1,2,3)$, stationarity gives $2x = \lambda a$, so $x = (\lambda/2)a$. Substituting into the constraint: $(\lambda/2)\lVert a \rVert^2 = 7\lambda = 14$, hence $\lambda = 2$ and $x^* = (1,2,3)$, $f^* = 14$.

Worked example 2: a curved constraint with four candidates

Optimize $f(x,y) = xy$ subject to $x^2 + y^2 = 2$. Here

$$ L = xy - \lambda(x^2 + y^2 - 2), $$

so $y - 2\lambda x = 0$, $x - 2\lambda y = 0$, and $x^2 + y^2 = 2$. Substitution is messy; multiply the first equation by $y$ and the second by $x$ to get $y^2 = 2\lambda xy = x^2$. At a solution $x$ and $y$ are both nonzero (if one is zero, both stationarity equations force the other to be zero, violating the constraint), so $x^2 = y^2$. Combined with the constraint, the candidates are $(1,1), (1,-1), (-1,1), (-1,-1)$.

Evaluating: $(1,1)$ and $(-1,-1)$ give $f = 1$ with $\lambda = 1/2$; $(1,-1)$ and $(-1,1)$ give $f = -1$ with $\lambda = -1/2$. On a circle the maximum and minimum both sit at stationary points, so stationarity alone did not classify them: comparing values did. Note also that $\lambda$ changes sign between the max and the min.

Reading $\lambda$ as a shadow price

Write the constraint as $g(x) = b$ and let $f^*(b)$ be the optimal value. Differentiating $L = f - \lambda(g - b)$ in $b$ at the optimum gives $df^*/db = \lambda$. In example 1, $f^*(b) = b^2/2$, so $df^*/db = b = 4$, matching $\lambda = 4$: relaxing the constraint by $\delta$ raises the optimal value by about $4\delta$. In example 2, with constraint $x^2 + y^2 = r^2$, the maximum is $r^2/2$, and its derivative with respect to the level $r^2$ is $1/2 = \lambda$. A zero multiplier means the constraint is not biting to first order.

Confirming the hand solution numerically

Checking gradients against finite differences is the habit from earlier modules; here you also verify that $\nabla f - \lambda \nabla g$ vanishes and that the solver reaches the same point.

import numpy as np
from scipy.optimize import minimize

# Example 1: min x^2 + y^2 s.t. x + y = 4
f = lambda v: v[0]**2 + v[1]**2
con = {"type": "eq", "fun": lambda v: v[0] + v[1] - 4}
res = minimize(f, x0=[0.0, 0.0], constraints=[con], method="SLSQP")

lam = 4.0
grad_f = np.array([2*res.x[0], 2*res.x[1]])
grad_g = np.array([1.0, 1.0])
print(res.x, res.fun)            # [2. 2.] 8.0
print(grad_f - lam*grad_g)       # ~ [0. 0.]

# Example 2: xy on the circle x^2 + y^2 = 2, parametrized scan
t = np.linspace(0.0, 2.0*np.pi, 400001)
vals = (np.sqrt(2)*np.cos(t)) * (np.sqrt(2)*np.sin(t))
print(vals.max(), vals.min())    # 1.0 -1.0

The scan is the honest check for example 2: it rules out any better feasible point that the stationarity system might have missed, including points where the multipliers fail.

Algebra habits that prevent sign errors

  • Fix one convention, $L = f - \lambda g$, and reuse it everywhere. Under that convention $\lambda = df^*/db$; the opposite sign in $L$ flips every interpretation.
  • Count equations before solving: one equality constraint in $n$ variables means $n+1$ equations, so there is no freedom left over.
  • When the system is nonlinear, prefer symmetry moves (multiply by a variable, subtract equations) over blind substitution.
  • Always test $\nabla g(x^*) \neq 0$ at your answer, and never skip comparing candidate values: stationarity is necessary, not sufficient.
  • Sanity-check $\lambda$ against a closed form or a finite-difference perturbation of the constraint level.

What to carry into descent methods

  • Constraints replace $\nabla f = 0$ with $\nabla f = \lambda \nabla g$: the gradient is absorbed by the surface normal.
  • Solving means solving the full system, including $\partial L/\partial \lambda = 0$, which is just the constraint restated.
  • $\lambda$ is a derivative, not decoration: it measures how much the optimal value moves when the constraint moves.

Next, gradient descent and learning rate behavior, where the Hessian from Module 2 tells you why a step size that works on one direction stalls or diverges on another.