Cheatsheet: Mathematics for Machine Learning 4: Build Core ML Algorithms from Scratch
Scope: Modules 1 to 3. Linear and cross-entropy models, PCA via SVD, k-means, GMM with EM, backprop MLP, gradient checking, cross-validation, and the library-baseline capstone. Everything in pure NumPy.
0. The Two Habits
- Derive on paper first. No NumPy line until the gradient (or update rule) is written and dimension-checked by hand.
- Keep the closed form as the reference answer. When a training loop misbehaves, compare against the analytic solution (normal equations, SVD) to isolate the bug.
Module 1: Linear and Cross-Entropy Models
1.1 Linear Regression
| Method | Formula / update | Use when |
|---|---|---|
| Normal equations | $\hat{w} = (X^\top X)^{-1} X^\top y$ | Small $d$, well-conditioned $X^\top X$; reference answer |
| Ridge-stable solve | $\hat{w} = (X^\top X + \lambda I)^{-1} X^\top y$ | $X^\top X$ near-singular / collinear features |
| Batch gradient descent | $w \leftarrow w - \eta \cdot \frac{2}{n} X^\top (Xw - y)$ | Large $n$, streaming, or as gradient-check target |
| Stochastic / mini-batch | Same update on a batch of size $B$ | $n$ large, memory-bound |
- Design matrix: prepend a column of ones for the bias, or keep
bseparate. Be consistent. - Loss: MSE $L = \frac{1}{n}|Xw - y|^2$. Gradient of $L$ w.r.t. $w$ is $\frac{2}{n}X^\top(Xw - y)$.
- Prefer
np.linalg.solve(A, b)overnp.linalg.inv(A) @ b(faster, more stable). - Center features before GD for faster convergence; scale for conditioning.
# Normal equation
w = np.linalg.solve(X.T @ X, X.T @ y)
# Gradient descent step
grad = (2.0 / n) * X.T @ (X @ w - y)
w -= eta * grad
1.2 Gradient Checking (Finite Difference Harness)
Central difference: $\frac{\partial L}{\partial \theta_i} \approx \frac{L(\theta + \epsilon e_i) - L(\theta - \epsilon e_i)}{2\epsilon}$
This cheatsheet is premium content. Sign in and purchase the course to view the full content. Sign in