# 🧮 Cheatsheet: Mathematics for Machine Learning 4: Build Core ML Algorithms from Scratch

By Better Wise

> This cheatsheet is premium content. Sign in and purchase the course to view the full content.

Scope: Modules 1 to 3. Linear and cross-entropy models, PCA via SVD, k-means, GMM with EM, backprop MLP, gradient checking, cross-validation, and the library-baseline capstone. Everything in pure NumPy.

## 0. The Two Habits

- **Derive on paper first.** No NumPy line until the gradient (or update rule) is written and dimension-checked by hand.
- **Keep the closed form as the reference answer.** When a training loop misbehaves, compare against the analytic solution (normal equations, SVD) to isolate the bug.

---

## Module 1: Linear and Cross-Entropy Models

### 1.1 Linear Regression

| Method | Formula / update | Use when |
| --- | --- | --- |
| Normal equations | $\hat{w} = (X^\top X)^{-1} X^\top y$ | Small $d$, well-conditioned $X^\top X$; reference answer |
| Ridge-stable solve | $\hat{w} = (X^\top X + \lambda I)^{-1} X^\top y$ | $X^\top X$ near-singular / collinear features |
| Batch gradient descent | $w \leftarrow w - \eta \cdot \frac{2}{n} X^\top (Xw - y)$ | Large $n$, streaming, or as gradient-check target |
| Stochastic / mini-batch | Same update on a batch of size $B$ | $n$ large, memory-bound |

- Design matrix: prepend a column of ones for the bias, or keep `b` separate. Be consistent.
- Loss: MSE $L = \frac{1}{n}\|Xw - y\|^2$. Gradient of $L$ w.r.t. $w$ is $\frac{2}{n}X^\top(Xw - y)$.
- Prefer `np.linalg.solve(A, b)` over `np.linalg.inv(A) @ b` (faster, more stable).
- Center features before GD for faster convergence; scale for conditioning.

```python
# Normal equation
w = np.linalg.solve(X.T @ X, X.T @ y)

# Gradient descent step
grad = (2.0 / n) * X.T @ (X @ w - y)
w -= eta * grad
```

### 1.2 Gradient Checking (Finite Difference Harness)

Central difference: $\frac{\partial L}{\partial \theta_i} \approx \frac{L(\theta + \epsilon e_i) - L(\theta - \epsilon e_i)}{2\epsilon}$