From partial derivatives to the gradient vector

By Better Wise

This lesson opens the module, and it is deliberately small: take a scalar function, differentiate it one coordinate at a time, stack the results into one vector, then prove each entry numerically. Jacobians, the chain rule, Hessians and gradient descent all reuse exactly this move.

By the end of this lesson you should be able to:

compute $\partial f/\partial x_i$ by freezing every other coordinate,
assemble $\nabla f(x)$ with the same shape as $x$,
read the gradient as a direction and a rate of change,
confirm each component with a central difference in Python.

Partial derivatives: one coordinate at a time

For $f: \mathbb{R}^n \to \mathbb{R}$, the partial derivative with respect to $x_i$ is the ordinary derivative of the one-variable slice you get by freezing the other coordinates:

$$ \frac{\partial f}{\partial x_i}(x) = \lim_{h \to 0} \frac{f(x_1, \dots, x_i + h, \dots, x_n) - f(x)}{h} $$

In practice: treat every other variable as a constant. In $f(x, y) = x^2 y + 3y^2$, differentiating in $x$ treats $y$ as a number, so $\partial f/\partial x = 2xy$; differentiating in $y$ treats $x$ as a number, so $\partial f/\partial y = x^2 + 6y$.

A partial derivative is a number once you plug in a point. It is a function only while the point stays symbolic.

Stacking the partials into the gradient

The gradient collects all $n$ partials into one object:

$$ \nabla f(x) = \begin{bmatrix} \partial f/\partial x_1 \ \vdots \ \partial f/\partial x_n \end{bmatrix} $$

Two facts worth memorizing: for a scalar-valued $f$, $\nabla f(x)$ has the same shape as $x$, and each entry answers "how fast does $f$ change if I nudge only this coordinate?" Partial derivatives are scalars; the gradient is a vector. Mixing the two up is the most common shape bug in optimization code.

Worked example: a product term and a square

$f(x,y) = x^2y + 3y^2$.

By hand: $\partial f/\partial x = 2xy$ and $\partial f/\partial y = x^2 + 6y$, so $\nabla f(x,y) = (2xy,; x^2 + 6y)$.

At $(1, 2)$: $\nabla f = (4, 13)$. At $(0, 0)$: $\nabla f = (0, 0)$, a stationary point; whether it is a minimum, maximum or saddle is a second-order question for Module 2.

Worked example: a quadratic with a cross term

$g(w_1, w_2) = (w_1 - 2)^2 + 3(w_2 + 1)^2 + w_1 w_2$.

Differentiate term by term, remembering that the cross term contributes to both partials:

$$ \frac{\partial g}{\partial w_1} = 2(w_1 - 2) + w_2, \qquad \frac{\partial g}{\partial w_2} = 6(w_2 + 1) + w_1 $$

At $(1, -1)$: $\nabla g = (2(-1) + (-1),; 6(0) + 1) = (-3, 1)$. This is the same style of quadratic we differentiate twice in Module 2, so keep the hand result.

What the gradient vector tells you

Step from $x$ by a small amount $t v$. The change in $f$ is approximately $t,\nabla f(x)\cdot v$, so the rate per unit step is that dot product. Maximizing over unit vectors $v$ gives $v = \nabla f/\lVert \nabla f \rVert$ with rate $\lVert \nabla f \rVert$:

$\nabla f(x)$ points in the direction of steepest increase of $f$ at $x$.
$\lVert \nabla f(x) \rVert$ is the slope in that direction.
$\nabla f(x)$ is orthogonal to the level set ${y : f(y) = f(x)}$ through $x$.

That orthogonality is why gradient steps cross contour lines at right angles, which is exactly what you see in descent plots.

Checking every component with a central difference

Hand algebra fails quietly, without an error message. The cure is a symmetric difference quotient per coordinate, with $e_i$ the $i$-th unit vector:

$$ \frac{\partial f}{\partial x_i}(x) \approx \frac{f(x + h e_i) - f(x - h e_i)}{2h}, \quad \text{error } O(h^2) $$

Central differences are second order, so each halving of $h$ shrinks truncation error by roughly 4x, until roundoff takes over.

import numpy as np

def f(x, y):
    return x**2 * y + 3.0 * y**2

def grad_hand(x, y):
    return np.array([2.0 * x * y, x**2 + 6.0 * y])

def grad_numeric(f, p, h=1e-6):
    g = np.zeros(len(p))
    for i in range(len(p)):
        e = np.zeros(len(p))
        e[i] = h
        g[i] = (f(*(p + e)) - f(*(p - e))) / (2 * h)
    return g

p = np.array([1.0, 2.0])
print(grad_hand(*p))                                             # [ 4. 13.]
print(grad_numeric(f, p))                                        # [ 4. 13.]
print(np.allclose(grad_hand(*p), grad_numeric(f, p), rtol=1e-6)) # True

Run the same check at $(0, 0)$ and at $(1, -1)$ for the second example. When a component disagrees, the bug is almost always a variable you thought you had frozen but did not, or a missing product-rule term.

Warning: Do not shrink $h$ toward 1e-16. In float64 the two function values become nearly identical, so cancellation error of order machine epsilon divided by $h$ dominates. Use $h$ between 1e-5 and 1e-7 and compare with np.allclose and a relative tolerance.

Habits that keep hand gradients honest

Freeze first, differentiate second: rewrite the function with the other variables replaced by constants, then differentiate.
Keep the shape straight: scalar $f$ means $\nabla f$ is a vector with $n$ entries, in the same argument order.
Check at a point where several terms vanish, then at a generic point; a single agreement can be luck.
Products, quotients and inner products each need one term per factor.
If the numeric check disagrees in one component only, suspect that coordinate's term; if all components disagree, suspect the point or the argument order.

Wrap-up

The gradient is the vector of partial derivatives, one entry per input coordinate.
It points toward steepest increase, its norm is that slope, and it is perpendicular to level sets.
A central-difference check per component is the default way to certify a hand-derived gradient.

Next lesson: Jacobians of vector-valued functions, where each output gets its own gradient row and the same numeric check runs output by output.

For a scalar function f from R^n to R, the gradient stacks the _____ into a single vector. The gradient of a scalar function has the _____, with one entry per coordinate. At each point, the gradient points in the direction of _____. The central-difference estimate of one gradient component is _____. A central difference has truncation error _____, so halving h cuts the error by about a factor of four.

Front: What does the partial derivative with respect to x_i measure? Back: The rate of change of f when only x_i moves, all other coordinates held fixed.

Front: How is the gradient built from partial derivatives? Back: It stacks all n partial derivatives into one vector, indexed like the input coordinates.

Front: What shape does the gradient of f: R^n to R have? Back: The same shape as x: an n-dimensional vector with one entry per input.

Front: Partial derivative of x^2 y + 3y^2 with respect to x? Back: 2xy

Front: Partial derivative of x^2 y + 3y^2 with respect to y? Back: x^2 + 6y

Front: Which direction does the gradient point? Back: The direction of steepest increase of f at that point.

Front: What is the slope of f along the gradient direction? Back: The norm of the gradient, ||grad f(x)||.

Front: How does the gradient relate to level sets? Back: It is orthogonal to the level set passing through the point.

Front: Central-difference formula for one gradient component? Back: (f(x + h e_i) - f(x - h e_i)) / (2h), with e_i the i-th unit vector.

Front: What is the truncation error order of a central difference? Back: O(h^2), second order.

Front: Why not use h = 1e-16 in float64? Back: Subtracting nearly equal numbers amplifies roundoff, which scales like machine epsilon over h.

Front: What does a zero gradient at a point tell you? Back: The point is stationary; classifying it needs second-order information.

What to carry forward

A gradient is not a new object to memorize: it is the partial derivatives you already know, stacked in the order of the inputs, so $\nabla f(\mathbf{x}) = \left(\partial f/\partial x_1, \dots, \partial f/\partial x_n\right)$.
It has the same shape as the input, which is the habit that will make Jacobians feel inevitable in the next lesson.
The gradient encodes geometry as well as numbers: it points uphill, its norm is the steepest slope, and it is orthogonal to the level set through the point.
Stationary points satisfy $\nabla f(\mathbf{x}) = 0$, but that condition alone never tells you whether you found a minimum, a maximum, or a saddle.
Every hand-derived gradient in this course gets a central-difference check. That single habit, (f(x + h e_i) - f(x - h e_i)) / (2h) with $h$ somewhere near $10^{-5}$ to $10^{-6}$ in float64, will catch sign errors, dropped terms, and misindexed components before they propagate into later modules.

You can now differentiate a scalar function of many variables and defend the answer numerically. The next lesson takes the same idea to vector-valued functions: when $f: \mathbb{R}^n \to \mathbb{R}^m$, the stack of partial derivatives becomes an $m \times n$ matrix, the Jacobian, and the shape rules you just learned are exactly what keep its rows and columns straight.

Partial derivative: Rate of change of f with respect to one coordinate while all others are held fixed.
Gradient: The vector of all partial derivatives of a scalar function, indexed like the inputs.
Stationary point: A point where the gradient is exactly zero.
Central difference: A symmetric finite-difference estimate of a derivative with O(h^2) truncation error.
Level set: The set of points where f takes one fixed value; the gradient is orthogonal to it.

Cards to run before the next lesson

Front: What is the partial derivative ∂f/∂x_i? Back: The rate of change of f along coordinate i while every other coordinate is held fixed.

Front: How is the gradient of a scalar function f: R^n → R written? Back: ∇f(x) = (∂f/∂x_1, ..., ∂f/∂x_n), a column vector with one entry per input coordinate.

Front: How many components does ∇f have when f: R^n → R? Back: Exactly n, one for each input coordinate; the shape tells you the input space.

Front: For f(x) = xᵀx = ||x||², what is ∇f? Back: 2x, because ∂f/∂x_i = 2x_i for every i.

Front: Which geometric direction does the gradient point? Back: Steepest ascent; minus the gradient is steepest descent.

Front: How does ∇f relate to a level set {x : f(x) = c}? Back: The gradient is orthogonal (normal) to the level set at each point on it.

Front: What condition defines a stationary point? Back: ∇f(x) = 0; on its own it does not distinguish minimum, maximum, or saddle.

Front: State the central-difference estimate of ∂f/∂x_i. Back: (f(x + h·e_i) − f(x − h·e_i)) / (2h), where e_i is the i-th unit vector.

Front: What is the truncation error of a central difference, and how does it compare to a forward difference? Back: O(h²) for central versus O(h) for forward, which is why central is the default check.

Front: What step size h is recommended for central differences in float64? Back: Roughly 1e-5 to 1e-6; large enough to avoid roundoff, small enough to keep truncation tiny.

Front: Why does an extremely small h make the numerical check worse? Back: Catastrophic cancellation: f(x + h·e_i) and f(x − h·e_i) become nearly equal floats and their difference loses significant digits.

Front: What agreement between analytic and central-difference gradients counts as a pass? Back: Relative error around 1e-6 or better; anything much larger points to a sign error, a dropped term, or a misindexed component.

Front: Which coordinate is perturbed when checking component i of the gradient? Back: Only the i-th one, via x + h·e_i; perturbing any other coordinate tests a different partial derivative.

Front: How do you read ∂f/∂x_i off a gradient computation at a point? Back: Evaluate ∇f at that point and take the i-th entry; the partial derivative is the component, not the whole vector.