Cheatsheet: Mathematics for Machine Learning 2: Calculus and Optimization for Training Models
Scope: Modules 1 to 3 (first-order derivatives, second-order behavior and convexity, constrained optimization and gradient descent). Every hand-derived result below is meant to be verified with a finite-difference check in Python.
0. The numeric check (use before trusting any derivative)
Central finite difference, error $O(h^2)$:
$$ \frac{\partial f}{\partial x_i} \approx \frac{f(x + h e_i) - f(x - h e_i)}{2h} $$
import numpy as np
def num_grad(f, x, h=1e-5):
x = np.asarray(x, dtype=float)
g = np.zeros_like(x)
for i in range(x.size):
e = np.zeros_like(x); e[i] = h
g[i] = (f(x + e) - f(x - e)) / (2 * h)
return g
# relative error check
def check(analytic, numeric, tol=1e-6):
err = np.linalg.norm(analytic - numeric) / (np.linalg.norm(numeric) + 1e-12)
assert err < tol, f"gradient mismatch, rel err = {err:.2e}"
return err
| Check | What it catches |
|---|---|
| Central difference vs analytic gradient | Sign errors, missing terms |
| Relative error < ~1e-6 with $h=10^{-5}$ | Confirms formula |
| $h$ too small (~1e-12) | Roundoff blows up the check |
| $h$ too large (~1e-2) | Truncation error hides bugs |
Tip: Jacobian numeric check: perturb one input coordinate at a time and compare the full output vector to the analytic row of $J$. Same code, one loop level deeper.
Module 1: First-order derivatives
1.1 Partial derivatives to the gradient vector
| Concept | Definition |
|---|---|
| Partial derivative | $\dfrac{\partial f}{\partial x_i} = \lim_{t\to 0}\dfrac{f(x+te_i)-f(x)}{t}$ |
| Gradient (scalar $f$) | $\nabla f(x) = \left(\frac{\partial f}{\partial x_1}, \dots, \frac{\partial f}{\partial x_n}\right)^\top$, same shape as $x$ |
| Directional derivative | $D_u f(x) = \nabla f(x)^\top u$ for unit $u$ |
| Steepest ascent | $u = \nabla f / |\nabla f|$ |
| Steepest descent | $u = -\nabla f / |\nabla f|$ |
Rules to have memorized:
This cheatsheet is premium content. Sign in and purchase the course to view the full content. Sign in