# 🧮 Jacobians of vector-valued functions

By Better Wise

> Sign in to view the full content of this lesson.

So far you have differentiated functions that output a single number, and collected their partial derivatives into a gradient vector. But most of the machinery you will build later (compositions, Hessians, gradients of losses with many outputs) involves functions that return **several numbers at once**. This lesson shows how to organize those derivatives into a matrix, the Jacobian, and how to check every entry numerically. Recall from the previous lesson: the gradient of $f:\mathbb{R}^n \to \mathbb{R}$ is the row vector of partials $\nabla f = \left[\frac{\partial f}{\partial x_1}, \dots, \frac{\partial f}{\partial x_n}\right]$. The Jacobian is simply that idea applied output by output.

By the end of this lesson you should be able to:

- Write a vector-valued function $F:\mathbb{R}^n \to \mathbb{R}^m$ as a stack of scalar components.
- Build the $m \times n$ Jacobian matrix $J_F$ entry by entry.
- Read each **row** of $J_F$ as the gradient of one output.
- Verify any Jacobian entry with a central finite difference in Python.
- Use the shape rule $m \times n$ to catch transposes and ordering errors.

## From one output to many

Write the function componentwise:

$$
F(x) = \begin{bmatrix} f_1(x) \\ \vdots \\ f_m(x) \end{bmatrix}, \qquad x \in \mathbb{R}^n .
$$

Each $f_i$ is a plain scalar function, so each has a gradient. Define

$$
J_F(x) = \begin{bmatrix} \dfrac{\partial f_1}{\partial x_1} & \cdots & \dfrac{\partial f_1}{\partial x_n} \\[4pt] \vdots & \ddots & \vdots \\[4pt] \dfrac{\partial f_m}{\partial x_1} & \cdots & \dfrac{\partial f_m}{\partial x_n} \end{bmatrix} = \begin{bmatrix} \nabla f_1(x)^\top \\ \vdots \\ \nabla f_m(x)^\top \end{bmatrix}.
$$

Two facts do all the bookkeeping for you:

1. **Shape is $m \times n$**, rows = outputs, columns = inputs.
2. **Row $i$ is the gradient of output $i$**, transposed to lie flat.

A useful sanity check: if $m = 1$ the Jacobian collapses to a single row, which is exactly the gradient of the previous lesson (as a row, not a column). The Jacobian is a strict generalization, not a new object.

### Worked example 1: an affine map

Let $F(x_1, x_2) = \begin{bmatrix} 3x_1 - 2x_2 \\ x_1 + 5x_2 \\ x_1 x_2 \end{bmatrix}$. Differentiate each row:

$$
J_F(x) = \begin{bmatrix} 3 & -2 \\ 1 & 5 \\ x_2 & x_1 \end{bmatrix}.
$$

Here $m = 3$, $n = 2$, so the shape is $3 \times 2$. Note the first two rows are constant because those outputs are linear; only the product output depends on $x$. Every entry is a partial derivative of exactly one output with respect to exactly one input, so each line above comes from holding the other variable fixed.

### Worked example 2: a polar-to-Cartesian style map

Let $G(r, \theta) = \begin{bmatrix} r\cos\theta \\ r\sin\theta \end{bmatrix}$, with $m = 2$, $n = 2$:

$$
J_G(r,\theta) = \begin{bmatrix} \cos\theta & -r\sin\theta \\ \sin\theta & r\cos\theta \end{bmatrix}.
$$

This matrix is the local linear approximation of the change of coordinates. Its determinant is $r$, which tells you how areas scale. You are not asked to interpret determinants here, only to notice that the rows are the two gradients:

$$
\nabla (r\cos\theta) = [\cos\theta,\ -r\sin\theta], \qquad \nabla (r\sin\theta) = [\sin\theta,\ r\cos\theta].
$$

### Worked example 3: a three-to-two map with a shared term

Let $H(x, y, z) = \begin{bmatrix} x^2 y + z \\ e^{xy} \end{bmatrix}$. Row by row:

$$
J_H = \begin{bmatrix} 2xy & x^2 & 1 \\ y e^{xy} & x e^{xy} & 0 \end{bmatrix}.
$$

The shape is $2 \times 3$: two outputs, three inputs. The zero in the bottom right is a real derivative, not a placeholder. The second output does not depend on $z$, so its partial with respect to $z$ vanishes. Missing zeros are the most common hand-derivation error in wide Jacobians.

## Why you need the shape rule

It is easy to write the transpose by reflex. A quick test on Worked Example 3: the input is a column vector in $\mathbb{R}^3$ and the output is in $\mathbb{R}^2$, so the linear map that best approximates $H$ near a point must send a 3-vector to a 2-vector. That forces the matrix to be $2 \times 3$. Multivariate derivative objects are linear maps between tangent spaces of matching sizes, and only the $m \times n$ layout multiplies correctly as $J_F \Delta x$.

## Verifying entries numerically

Never trust a hand-derived Jacobian until a number agrees. The central difference for a single entry is

$$
\frac{\partial f_i}{\partial x_j} \approx \frac{f_i(x + h e_j) - f_i(x - h e_j)}{2h},
$$

with error $O(h^2)$. Take $h = 10^{-6}$; you should see agreement to around 6 to 9 significant digits.

```python
import numpy as np

def H(x):
    x1, x2, x3 = x
    return np.array([x1**2 * x2 + x3, np.exp(x1 * x2)])

def jacobian_numeric(f, x, h=1e-6):
    x = np.asarray(x, dtype=float)
    n = x.size
    m = f(x).size
    J = np.zeros((m, n))
    for j in range(n):
        e = np.zeros(n); e[j] = h
        J[:, j] = (f(x + e) - f(x - e)) / (2 * h)
    return J

def jacobian_hand(x, y, z):
    return np.array([[2*x*y, x**2, 1.0],
                     [y*np.exp(x*y), x*np.exp(x*y), 0.0]])

x0 = np.array([0.7, -1.3, 2.1])
print(np.allclose(jacobian_hand(*x0), jacobian_numeric(H, x0), atol=1e-6))
```

The output is `True`. If it is `False`, always print the two matrices side by side. A single mismatched entry almost always means one partial was differentiated with respect to the wrong variable, or a factor was dropped.

> **Important:** 
> Check the numeric routine itself on a case you know analytically first. For $G(r,\theta)$ above, `jacobian_numeric` should return $\begin{bmatrix} \cos\theta & -r\sin\theta \\ \sin\theta & r\cos\theta \end{bmatrix}$. If your checker is wrong, it will happily confirm a wrong Jacobian.

A short derivation of the exponential row, to keep the habit honest: $\frac{\partial}{\partial x} e^{xy} = y e^{xy}$ because $xy$ has $\partial / \partial x = y$ and the chain rule contributes the outer derivative $e^{xy}$. Then $\frac{\partial}{\partial y} e^{xy} = x e^{xy}$ by the same reasoning with the factor swapped.

### Practice

Derive $J$ for $F(x_1,x_2) = \begin{bmatrix} x_1^3 x_2 \\ \log(x_1 + x_2^2) \\ x_1 - x_2 \end{bmatrix}$, then verify with finite differences at $x = (1.5, 0.8)$.

**Click to reveal the answer**

$$
J_F = \begin{bmatrix} 3x_1^2 x_2 & x_1^3 \\ \dfrac{1}{x_1 + x_2^2} & \dfrac{2x_2}{x_1 + x_2^2} \\ 1 & -1 \end{bmatrix}, \quad J_F(1.5, 0.8) \approx \begin{bmatrix} 5.4 & 3.375 \\ 0.4591 & 0.4897 \\ 1 & -1 \end{bmatrix}.
$$

## Field notes for assembling Jacobians

- **Write the shape before you differentiate.** Annotate "$m \times n$" at the top of your page. Half of all transpose errors die at this step.
- **Never skip zero rows or columns.** Forgetting that an output does not depend on an input is the top source of wrong Jacobians in wide maps.
- **Check one row against one gradient.** If row $i$ differs from $\nabla f_i^\top$, you have a definition error, not a typo.
- **Pick an awkward point for the numeric check.** At $x=(0,0)$ many Jacobians are trivial or singular; random points expose real mistakes.
- **Do not shrink $h$ blindly.** Below about $10^{-8}$ the subtraction in the numerator loses accuracy and the error grows; $10^{-5}$ to $10^{-7}$ is the sweet spot.
- **Log-scale comparison.** If an entry is huge, compare relative error, not absolute tolerance, otherwise `allclose` will mislead you.

## Closing

The Jacobian is the gradient idea extended across outputs: an $m \times n$ block whose rows are the output gradients and whose columns are the input directions. Build it componentwise, check the shape, then confirm entries with a central difference. Cloze check: in a Jacobian, _____ index outputs while _____ index inputs; the entry $J_{ij}$ equals _____.

Next up: the **chain rule for composed functions**, where Jacobians multiply, and the shape rule above becomes your fastest error detector.

Front: What is the shape of the Jacobian of F: R^n -> R^m?
Back: m x n: one row per output, one column per input.

Front: What does row i of J_F equal?
Back: The transposed gradient of output f_i.

Front: What does column j of J_F collect?
Back: The partial derivatives of every output with respect to input x_j.

Front: What does J_F collapse to when m = 1?
Back: A single row, which is the gradient of the scalar function.

Front: Formula for the central-difference approximation of one Jacobian entry?
Back: (f_i(x + h e_j) - f_i(x - h e_j)) / (2h), with error O(h^2).

Front: Recommended step size h for a central-difference Jacobian check?
Back: Around 1e-5 to 1e-7; much smaller loses precision to cancellation.

Front: Why must an output that ignores an input still appear in J_F?
Back: Its row keeps an explicit zero in that column; dropping it is a derivation error.

Front: For G(r, theta) = (r cos theta, r sin theta), what is J_G?
Back: [[cos theta, -r sin theta], [sin theta, r cos theta]].

Front: What is the Jacobian of an affine map A x + b?
Back: The constant matrix A, since all entries are constants.

Front: How do you interpret J_F locally?
Back: As the matrix of the best linear approximation of F near a point.

Front: Which layout multiplies correctly as J_F times a step vector?
Back: The m x n layout, sending an n-vector step to an m-vector output change.

Front: Where is the classic place a reordered or transposed Jacobian is caught?
Back: The shape check, before any differentiation is verified.

In a Jacobian of F from R^n to R^m, _____ index outputs while _____ index inputs.

The entry J_ij equals _____.

If an output does not depend on an input, the corresponding Jacobian entry is _____, not omitted.

A central finite difference approximates a Jacobian entry with error of order _____.

---

## Contents

Part of [🧮 Mathematics for Machine Learning 2: Calculus and Optimization for Training Models](https://everwise.study/courses/mathematics-for-machine-learning-2-calculus-and-optimization-for-training.md)

- **📐 First-order derivatives: gradients, Jacobians and the chain rule**
  - [📐 From partial derivatives to the gradient vector](https://everwise.study/courses/mathematics-for-machine-learning-2-calculus-and-optimization-for-training/partial-derivatives-to-gradient-vector.md)
  - [🧮 Jacobians of vector-valued functions](https://everwise.study/courses/mathematics-for-machine-learning-2-calculus-and-optimization-for-training/jacobian-vector-valued-functions.md) (sign-in required) — this lesson
  - [🔗 Chain rule for composed functions](https://everwise.study/courses/mathematics-for-machine-learning-2-calculus-and-optimization-for-training/chain-rule-composed-functions.md)
  - [✅ Checkpoint: first-order derivatives](https://everwise.study/courses/mathematics-for-machine-learning-2-calculus-and-optimization-for-training/checkpoint-first-order-derivatives.md)
- **📉 Second-order behavior and convexity**
  - [🌀 Hessians and second-order behavior](https://everwise.study/courses/mathematics-for-machine-learning-2-calculus-and-optimization-for-training/hessians-second-order-behavior.md)
  - [📈 Taylor expansions, convexity and local minima](https://everwise.study/courses/mathematics-for-machine-learning-2-calculus-and-optimization-for-training/taylor-expansions-convexity-local-minima.md)
  - [✅ Checkpoint: curvature and convexity](https://everwise.study/courses/mathematics-for-machine-learning-2-calculus-and-optimization-for-training/checkpoint-curvature-and-convexity.md)
- **⛰️ Constrained optimization and gradient descent**
  - [⚖️ Lagrange multipliers for constrained problems](https://everwise.study/courses/mathematics-for-machine-learning-2-calculus-and-optimization-for-training/lagrange-multipliers-constrained-problems.md)
  - [🔽 Gradient descent and learning rate behavior](https://everwise.study/courses/mathematics-for-machine-learning-2-calculus-and-optimization-for-training/gradient-descent-learning-rate-behavior.md) (premium content)
  - [✅ Checkpoint: constrained optimization and descent](https://everwise.study/courses/mathematics-for-machine-learning-2-calculus-and-optimization-for-training/checkpoint-constrained-optimization-descent.md) (premium content)
- [Course cheatsheet](https://everwise.study/courses/mathematics-for-machine-learning-2-calculus-and-optimization-for-training/cheatsheet.md) (premium content)