# 🧮 Cheatsheet: Mathematics for Machine Learning 2: Calculus and Optimization for Training Models

By Better Wise

> This cheatsheet is premium content. Sign in and purchase the course to view the full content.

Scope: Modules 1 to 3 (first-order derivatives, second-order behavior and convexity, constrained optimization and gradient descent). Every hand-derived result below is meant to be verified with a finite-difference check in Python.

---

## 0. The numeric check (use before trusting any derivative)

Central finite difference, error $O(h^2)$:

$$
\frac{\partial f}{\partial x_i} \approx \frac{f(x + h e_i) - f(x - h e_i)}{2h}
$$

```python
import numpy as np

def num_grad(f, x, h=1e-5):
    x = np.asarray(x, dtype=float)
    g = np.zeros_like(x)
    for i in range(x.size):
        e = np.zeros_like(x); e[i] = h
        g[i] = (f(x + e) - f(x - e)) / (2 * h)
    return g

# relative error check
def check(analytic, numeric, tol=1e-6):
    err = np.linalg.norm(analytic - numeric) / (np.linalg.norm(numeric) + 1e-12)
    assert err < tol, f"gradient mismatch, rel err = {err:.2e}"
    return err
```

| Check | What it catches |
| --- | --- |
| Central difference vs analytic gradient | Sign errors, missing terms |
| Relative error < ~1e-6 with $h=10^{-5}$ | Confirms formula |
| $h$ too small (~1e-12) | Roundoff blows up the check |
| $h$ too large (~1e-2) | Truncation error hides bugs |

> **Tip:** 
> Jacobian numeric check: perturb one input coordinate at a time and compare the full output vector to the analytic row of $J$. Same code, one loop level deeper.

---

## Module 1: First-order derivatives

### 1.1 Partial derivatives to the gradient vector

| Concept | Definition |
| --- | --- |
| Partial derivative | $\dfrac{\partial f}{\partial x_i} = \lim_{t\to 0}\dfrac{f(x+te_i)-f(x)}{t}$ |
| Gradient (scalar $f$) | $\nabla f(x) = \left(\frac{\partial f}{\partial x_1}, \dots, \frac{\partial f}{\partial x_n}\right)^\top$, same shape as $x$ |
| Directional derivative | $D_u f(x) = \nabla f(x)^\top u$ for unit $u$ |
| Steepest ascent | $u = \nabla f / \|\nabla f\|$ |
| Steepest descent | $u = -\nabla f / \|\nabla f\|$ |

Rules to have memorized: