Mathematics for Machine Learning 5: Information Theory, Matrix Methods and Numerical Stability

What you will achieve

By the end of this course you will be able to look at a training loss and say what it is optimizing in information-theoretic terms, then predict how that choice will fail. Concretely, you will be able to:

• Derive entropy, cross-entropy and KL divergence as one family of losses, and explain when each is the right objective.
• Read a contrastive objective such as InfoNCE as a mutual information bound, and see what the bound does and does not guarantee.
• Predict the behavioral difference between forward and reverse KL: which one covers modes, which one collapses onto one, and why that shows up as specific training failures.
• Solve and regularize linear systems with QR, Cholesky and the full SVD, and choose between them based on cost, stability and rank.
• Diagnose ill-conditioning, explain why a system is ill-posed, and apply Tikhonov regularization with a defensible choice of the damping term.
• Recognize catastrophic cancellation in a floating-point pipeline, restructure the computation to avoid it, and reason about what mixed precision changes and what it silently breaks.
• Use an eigenbasis to filter and smooth signals, and connect that operation to the spectral view of a linear operator.
• Read the ELBO as a KL objective and explain the variational gap as a trade between fit and closeness to the prior.

The through-line is a research scientist's habit: treat floating-point error as a first-class topic, and know exactly where real models break rather than where the textbook says they should.

Who this course is for

This is an advanced course. It assumes you are already fluent in the mathematical and numerical tools it builds on, so those are not re-taught:

• Linear algebra, including the SVD and what its factors mean.
• Probability and expectation, including conditional expectation and basic distributions.
• Calculus and gradients, including the chain rule and gradients of scalar functions.
• Working NumPy skills: you can write array code, broadcast, and read a stack trace from a shape mismatch.

If you can read a matrix factorization and a gradient expression without pausing, you have the background. The course spends its time on how these tools behave in practice, not on re-deriving them. Expect to write and reason about small numerical examples in your head or in a scratch notebook; there are no labs, but the theory lessons carry worked examples you are meant to follow line by line.

Course roadmap

Module 1: Information-Theoretic Training Objectives

• 1.1 Entropy, Cross-Entropy and KL: One Family of Losses. Shows how the three quantities relate, so that a cross-entropy loss and a KL penalty are recognized as the same object viewed from different angles.
• 1.2 Mutual Information as an Objective: InfoNCE and Contrastive Bounds. Derives the contrastive bound and explains what the temperature and batch size are actually controlling.
• 1.3 Forward vs Reverse KL: Mode Covering, Mode Seeking and Failure Modes. Compares the two directions and connects each to the failure mode it produces in a trained model.
• 1.4 Checkpoint: Information-Theoretic Objectives. Consolidates the module and tests whether you can pick the right objective for a stated problem.

Module 2: Solving and Regularizing Linear Systems

• 2.1 QR and Cholesky: Factorizations That Solve. Covers the two workhorse factorizations, when each applies, and what each costs.
• 2.2 Full SVD: Pseudoinverse, Truncation and Rank Control. Uses the SVD to solve rank-deficient problems and to deliberately truncate a system down to a stable rank.
• 2.3 Conditioning, Ill-Posed Problems and Tikhonov Regularization. Explains why a small residual does not mean a good answer, and how to choose regularization that stabilizes without destroying the signal.
• 2.4 Checkpoint: Linear Systems in Practice. Tests factorization choice and regularization judgment on realistic setups.

Module 3: Floating Point, Spectral Methods and Variational Inference

• 3.1 Floating Point, Catastrophic Cancellation and Mixed Precision. Walks through where precision is lost, how to spot cancellation, and what mixed precision changes under the hood.
• 3.2 Spectral Methods: Eigenbases, Filtering and Smoothing. Builds the eigenbasis view and uses it to design filters and smoothing operators.
• 3.3 First Look at Variational Inference: The ELBO as a KL Objective. Presents the ELBO as a KL objective and reads the variational gap as a fit-versus-prior trade.
• 3.4 Checkpoint: Precision, Spectra and Variational Bounds. Closes the course by testing whether you can connect numerical precision, spectral structure and the variational bound into one picture.

How the course works

The course runs about 320 minutes across 12 lessons, roughly 25 to 30 minutes per lesson. Each module has three theory lessons and one checkpoint.

• Theory lessons carry the core material as short explanations, visuals and worked examples in depth. The examples are the practice: you are meant to follow the arithmetic, predict the next step, and check yourself against the derivation.
• Checkpoint lessons are quizzes that test judgment, not recall. They ask you to choose an objective, a factorization or a regularization strength and justify it.
• No labs or projects. This is a deep-dive reading course. The work happens in the examples and in your own reasoning.

You can take it linearly, and for a first pass you should: Module 1 sets up the objectives, Module 2 sets up the linear algebra machinery, and Module 3 combines them with numerical precision and the variational bound. The modules are loosely coupled, so if you already know the information-theoretic objectives cold, you can start at Module 2 and return to 1.3 for the forward-versus-reverse KL contrast, which the later lessons lean on. Do not skip 3.1: the floating-point material is the spine of the course, and the later examples assume it.