Jacobians of vector-valued functions

A useful sanity check: if $m = 1$ the Jacobian collapses to a single row, which is exactly the gradient of the previous lesson (as a row, not a column). The Jacobian is a strict generalization, not a new object.

Worked example 1: an affine map

Let $F(x_1, x_2) = \begin{bmatrix} 3x_1 - 2x_2 \ x_1 + 5x_2 \ x_1 x_2 \end{bmatrix}$. Differentiate each row:

$$ J_F(x) = \begin{bmatrix} 3 & -2 \ 1 & 5 \ x_2 & x_1 \end{bmatrix}. $$

Here $m = 3$, $n = 2$, so the shape is $3 \times 2$. Note the first two rows are constant because those outputs are linear; only the product output depends on $x$. Every entry is a partial derivative of exactly one output with respect to exactly one input, so each line above comes from holding the other variable fixed.

Worked example 2: a polar-to-Cartesian style map

Let $G(r, \theta) = \begin{bmatrix} r\cos\theta \ r\sin\theta \end{bmatrix}$, with $m = 2$, $n = 2$:

$$ J_G(r,\theta) = \begin{bmatrix} \cos\theta & -r\sin\theta \ \sin\theta & r\cos\theta \end{bmatrix}. $$

This matrix is the local linear approximation of the change of coordinates. Its determinant is $r$, which tells you how areas scale. You are not asked to interpret determinants here, only to notice that the rows are the two gradients:

$$ \nabla (r\cos\theta) = [\cos\theta,\ -r\sin\theta], \qquad \nabla (r\sin\theta) = [\sin\theta,\ r\cos\theta]. $$

Worked example 3: a three-to-two map with a shared term

Let $H(x, y, z) = \begin{bmatrix} x^2 y + z \ e^{xy} \end{bmatrix}$. Row by row:

$$ J_H = \begin{bmatrix} 2xy & x^2 & 1 \ y e^{xy} & x e^{xy} & 0 \end{bmatrix}. $$

The shape is $2 \times 3$: two outputs, three inputs. The zero in the bottom right is a real derivative, not a placeholder. The second output does not depend on $z$, so its partial with respect to $z$ vanishes. Missing zeros are the most common hand-derivation error in wide Jacobians.

Why you need the shape rule

It is easy to write the transpose by reflex. A quick test on Worked Example 3: the input is a column vector in $\mathbb{R}^3$ and the output is in $\mathbb{R}^2$, so the linear map that best approximates $H$ near a point must send a 3-vector to a 2-vector. That forces the matrix to be $2 \times 3$. Multivariate derivative objects are linear maps between tangent spaces of matching sizes, and only the $m \times n$ layout multiplies correctly as $J_F \Delta x$.

Verifying entries numerically

Never trust a hand-derived Jacobian until a number agrees. The central difference for a single entry is

$$ \frac{\partial f_i}{\partial x_j} \approx \frac{f_i(x + h e_j) - f_i(x - h e_j)}{2h}, $$

with error $O(h^2)$. Take $h = 10^{-6}$; you should see agreement to around 6 to 9 significant digits.

import numpy as np

def H(x):
    x1, x2, x3 = x
    return np.array([x1**2 * x2 + x3, np.exp(x1 * x2)])

def jacobian_numeric(f, x, h=1e-6):
    x = np.asarray(x, dtype=float)
    n = x.size
    m = f(x).size
    J = np.zeros((m, n))
    for j in range(n):
        e = np.zeros(n); e[j] = h
        J[:, j] = (f(x + e) - f(x - e)) / (2 * h)
    return J

def jacobian_hand(x, y, z):
    return np.array([[2*x*y, x**2, 1.0],
                     [y*np.exp(x*y), x*np.exp(x*y), 0.0]])

x0 = np.array([0.7, -1.3, 2.1])
print(np.allclose(jacobian_hand(*x0), jacobian_numeric(H, x0), atol=1e-6))

The output is True. If it is False, always print the two matrices side by side. A single mismatched entry almost always means one partial was differentiated with respect to the wrong variable, or a factor was dropped.

Important: Check the numeric routine itself on a case you know analytically first. For $G(r,\theta)$ above, jacobian_numeric should return $\begin{bmatrix} \cos\theta & -r\sin\theta \ \sin\theta & r\cos\theta \end{bmatrix}$. If your checker is wrong, it will happily confirm a wrong Jacobian.

A short derivation of the exponential row, to keep the habit honest: $\frac{\partial}{\partial x} e^{xy} = y e^{xy}$ because $xy$ has $\partial / \partial x = y$ and the chain rule contributes the outer derivative $e^{xy}$. Then $\frac{\partial}{\partial y} e^{xy} = x e^{xy}$ by the same reasoning with the factor swapped.

Practice

Derive $J$ for $F(x_1,x_2) = \begin{bmatrix} x_1^3 x_2 \ \log(x_1 + x_2^2) \ x_1 - x_2 \end{bmatrix}$, then verify with finite differences at $x = (1.5, 0.8)$.

Click to reveal the answer

$$ J_F = \begin{bmatrix} 3x_1^2 x_2 & x_1^3 \ \dfrac{1}{x_1 + x_2^2} & \dfrac{2x_2}{x_1 + x_2^2} \ 1 & -1 \end{bmatrix}, \quad J_F(1.5, 0.8) \approx \begin{bmatrix} 5.4 & 3.375 \ 0.4591 & 0.4897 \ 1 & -1 \end{bmatrix}. $$

Field notes for assembling Jacobians

• Write the shape before you differentiate. Annotate "$m \times n$" at the top of your page. Half of all transpose errors die at this step.
• Never skip zero rows or columns. Forgetting that an output does not depend on an input is the top source of wrong Jacobians in wide maps.
• Check one row against one gradient. If row $i$ differs from $\nabla f_i^\top$, you have a definition error, not a typo.
• Pick an awkward point for the numeric check. At $x=(0,0)$ many Jacobians are trivial or singular; random points expose real mistakes.
• Do not shrink $h$ blindly. Below about $10^{-8}$ the subtraction in the numerator loses accuracy and the error grows; $10^{-5}$ to $10^{-7}$ is the sweet spot.
• Log-scale comparison. If an entry is huge, compare relative error, not absolute tolerance, otherwise allclose will mislead you.

Closing

The Jacobian is the gradient idea extended across outputs: an $m \times n$ block whose rows are the output gradients and whose columns are the input directions. Build it componentwise, check the shape, then confirm entries with a central difference. Cloze check: in a Jacobian, _____ index outputs while _____ index inputs; the entry $J_{ij}$ equals _____.

Next up: the chain rule for composed functions, where Jacobians multiply, and the shape rule above becomes your fastest error detector.

Front: What is the shape of the Jacobian of F: R^n -> R^m? Back: m x n: one row per output, one column per input.

Front: What does row i of J_F equal? Back: The transposed gradient of output f_i.

Front: What does column j of J_F collect? Back: The partial derivatives of every output with respect to input x_j.

Front: What does J_F collapse to when m = 1? Back: A single row, which is the gradient of the scalar function.

Front: Formula for the central-difference approximation of one Jacobian entry? Back: (f_i(x + h e_j) - f_i(x - h e_j)) / (2h), with error O(h^2).

Front: Recommended step size h for a central-difference Jacobian check? Back: Around 1e-5 to 1e-7; much smaller loses precision to cancellation.

Front: Why must an output that ignores an input still appear in J_F? Back: Its row keeps an explicit zero in that column; dropping it is a derivation error.

Front: For G(r, theta) = (r cos theta, r sin theta), what is J_G? Back: [[cos theta, -r sin theta], [sin theta, r cos theta]].

Front: What is the Jacobian of an affine map A x + b? Back: The constant matrix A, since all entries are constants.

Front: How do you interpret J_F locally? Back: As the matrix of the best linear approximation of F near a point.

Front: Which layout multiplies correctly as J_F times a step vector? Back: The m x n layout, sending an n-vector step to an m-vector output change.

Front: Where is the classic place a reordered or transposed Jacobian is caught? Back: The shape check, before any differentiation is verified.

In a Jacobian of F from R^n to R^m, _____ index outputs while _____ index inputs.

The entry J_ij equals _____.

If an output does not depend on an input, the corresponding Jacobian entry is _____, not omitted.

A central finite difference approximates a Jacobian entry with error of order _____.