Name: Mathematics for Machine Learning 4: Build Core ML Algorithms from Scratch
Author: Better Wise

Turn three courses of math into working code: derive and implement regression, PCA, clustering, EM and backpropagation from scratch, finishing with a project that benchmarks your models.

What you will achieve

By the end of this course you will be able to build the core of a classical machine learning stack in pure NumPy, with no scikit-learn or scipy.optimize doing the hard work for you. Concretely, you will be able to:

Derive the normal equations and the gradient descent update for linear regression, and implement both.
Write a finite-difference gradient checker and use it to prove your analytic gradients are correct before trusting any training run.
Derive cross-entropy loss for binary logistic regression and extend it to the softmax classifier, handling the numerical stability traps that break naive implementations.
Implement PCA via SVD, read off explained variance, and choose the number of components with a defensible rule.
Implement k-means with Lloyd's algorithm and k-means++ initialization, and explain why initialization changes the result.
Fit a Gaussian mixture model with expectation maximization and compare its soft assignments against k-means.
Derive and implement backpropagation for a one-hidden-layer MLP, then verify it with your gradient checker.
Run cross-validation and a fair baseline comparison protocol, so your results are a claim you can defend, not a number that happened to look good.

The through-line is a habit: derive the gradient on paper, check it numerically, then implement it.

Who this course is for

This is an advanced course. It assumes you already have:

Linear algebra: matrices, eigenvalues, SVD.
Calculus: gradients and the chain rule.
Probability: distributions and maximum likelihood.
Python and NumPy.

We do not re-teach any of these. If the chain rule across a matrix product still feels shaky, the backpropagation module will be rough. The course is for people who want to understand these algorithms at the level of their equations and their failure modes, not just call them from a library.

Course roadmap

Module 1: Linear and Cross-Entropy Models from Scratch

1.1 Linear Regression with Normal Equations and Gradient Descent: derive both solutions, then implement them and compare.
1.2 Gradient Checking with a Finite-Difference Harness: build the harness you will reuse for the rest of the course.
1.3 Logistic Regression to Softmax: Cross-Entropy Done Right: derive the loss and gradient, then extend to multiclass with numerically stable softmax.
1.4 Checkpoint: Linear and Cross-Entropy Models: a quiz to consolidate the module before moving on.

Module 2: Unsupervised Learning from Scratch

2.1 PCA via SVD: Explained Variance and Component Selection: implement PCA, interpret the spectrum, and pick components.
2.2 k-means: Lloyd's Algorithm and k-means++ Initialization: implement both and see the difference in practice.
2.3 Gaussian Mixture Models with Expectation Maximization: derive the E and M steps and fit a GMM.
2.4 Checkpoint: PCA, k-means and GMM: a quiz covering the unsupervised module.

Module 3: Backpropagation and Benchmark Capstone

3.1 Backpropagation for a One-Hidden-Layer MLP: derive the backward pass by hand, then implement and gradient-check it.
3.2 Cross-Validation and a Fair Baseline Comparison Protocol: set up folds, metrics, and a protocol that treats your implementation and the library on equal footing.
3.3 Capstone: NumPy Implementations vs Library Baselines: run the full comparison and write up the results.
3.4 Checkpoint: Backpropagation and Model Comparison: the final quiz.

How the course works

The course runs about 340 minutes across 12 lessons. That is a working estimate, not a deadline. Expect to spend more time on the derivation-heavy lessons (1.2, 1.3, 3.1) than on the quiz checkpoints.

Lessons come in three formats:

Theory and derivation. The equations come first. You derive the gradient before you are allowed to write the NumPy.
Practice. Exercises where you implement what you just derived, usually paired with a numerical check or a small experiment.
Checkpoints. Short quizzes at the end of each module (1.4, 2.4, 3.4) to confirm the concepts landed.

You can take the course linearly, and for most learners that is the right call: the gradient checker from 1.2 is reused in 3.1, and the cross-validation protocol from 3.2 is reused in the capstone. Skipping ahead is possible if you already own a topic. If you do skip, do not skip 1.2 or 3.2, because the later work depends on them.

The practice style is exercises and a project. There is no passive viewing path: you will be writing code and checking it against finite differences throughout.

The course project

The project is the NumPy vs Library Benchmark, one continuous thread that runs from lesson 1.1 to the capstone in 3.3. It is not a set of disconnected exercises. You build it incrementally:

1.1 and 1.3 give you the linear and softmax classifiers, plus their gradients.
1.2 gives you the gradient checker that validates every model you write.
2.1 through 2.3 give you the unsupervised side: PCA, k-means, and GMM.
3.1 gives you the MLP and its backward pass.
3.2 gives you the protocol: cross-validation folds, metrics, and a fair way to compare your code to a library baseline.
3.3 is where it all comes together: you run your implementations against the library baselines under the same protocol and write up where they match, where they diverge, and why.

By the end you will have a small, honest, from-scratch ML library and a benchmark report you can show. The value is not that your code beats scikit-learn, it is that you can explain every line of it and defend every number in the comparison.