# 🎲 Mathematics for Machine Learning 5: Information Theory, Matrix Methods and Numerical Stability

By Better Wise · Generated by Everwise

Go deeper into the math that separates working models from fragile ones: information-theoretic losses, stable matrix decompositions, and floating-point pitfalls demonstrated on real computations.

- **Level:** Advanced
- **Goal:** Deep dive
- **Language:** English
- **Focus:** [information theory](/courses?q=information%20theory), [matrix decompositions](/courses?q=matrix%20decompositions), [numerical stability](/courses?q=numerical%20stability), [spectral methods](/courses?q=spectral%20methods), [variational inference basics](/courses?q=variational%20inference%20basics)
- **Prior knowledge:** Linear algebra including SVD, Probability and expectation, Calculus and gradients, Working NumPy skills
- **Practice:** In-depth examples
- **Time:** ~5.3h
- **Voice:** a research scientist who treats floating-point error as a first-class topic and shows exactly where real models silently break
- **Not covered:** Measure theory, Functional analysis, Production MLOps

## What you will achieve

By the end of this course you will be able to look at a training loss and say what it is optimizing in information-theoretic terms, then predict how that choice will fail. Concretely, you will be able to:

- Derive entropy, cross-entropy and KL divergence as one family of losses, and explain when each is the right objective.
- Read a contrastive objective such as InfoNCE as a mutual information bound, and see what the bound does and does not guarantee.
- Predict the behavioral difference between forward and reverse KL: which one covers modes, which one collapses onto one, and why that shows up as specific training failures.
- Solve and regularize linear systems with QR, Cholesky and the full SVD, and choose between them based on cost, stability and rank.
- Diagnose ill-conditioning, explain why a system is ill-posed, and apply Tikhonov regularization with a defensible choice of the damping term.
- Recognize catastrophic cancellation in a floating-point pipeline, restructure the computation to avoid it, and reason about what mixed precision changes and what it silently breaks.
- Use an eigenbasis to filter and smooth signals, and connect that operation to the spectral view of a linear operator.
- Read the ELBO as a KL objective and explain the variational gap as a trade between fit and closeness to the prior.

The through-line is a research scientist's habit: treat floating-point error as a first-class topic, and know exactly where real models break rather than where the textbook says they should.

## Who this course is for

This is an advanced course. It assumes you are already fluent in the mathematical and numerical tools it builds on, so those are not re-taught:

- Linear algebra, including the SVD and what its factors mean.
- Probability and expectation, including conditional expectation and basic distributions.
- Calculus and gradients, including the chain rule and gradients of scalar functions.
- Working NumPy skills: you can write array code, broadcast, and read a stack trace from a shape mismatch.

If you can read a matrix factorization and a gradient expression without pausing, you have the background. The course spends its time on how these tools behave in practice, not on re-deriving them. Expect to write and reason about small numerical examples in your head or in a scratch notebook; there are no labs, but the theory lessons carry worked examples you are meant to follow line by line.

## Course roadmap

**Module 1: Information-Theoretic Training Objectives**

- **1.1 Entropy, Cross-Entropy and KL: One Family of Losses.** Shows how the three quantities relate, so that a cross-entropy loss and a KL penalty are recognized as the same object viewed from different angles.
- **1.2 Mutual Information as an Objective: InfoNCE and Contrastive Bounds.** Derives the contrastive bound and explains what the temperature and batch size are actually controlling.
- **1.3 Forward vs Reverse KL: Mode Covering, Mode Seeking and Failure Modes.** Compares the two directions and connects each to the failure mode it produces in a trained model.
- **1.4 Checkpoint: Information-Theoretic Objectives.** Consolidates the module and tests whether you can pick the right objective for a stated problem.

**Module 2: Solving and Regularizing Linear Systems**

- **2.1 QR and Cholesky: Factorizations That Solve.** Covers the two workhorse factorizations, when each applies, and what each costs.
- **2.2 Full SVD: Pseudoinverse, Truncation and Rank Control.** Uses the SVD to solve rank-deficient problems and to deliberately truncate a system down to a stable rank.
- **2.3 Conditioning, Ill-Posed Problems and Tikhonov Regularization.** Explains why a small residual does not mean a good answer, and how to choose regularization that stabilizes without destroying the signal.
- **2.4 Checkpoint: Linear Systems in Practice.** Tests factorization choice and regularization judgment on realistic setups.

**Module 3: Floating Point, Spectral Methods and Variational Inference**

- **3.1 Floating Point, Catastrophic Cancellation and Mixed Precision.** Walks through where precision is lost, how to spot cancellation, and what mixed precision changes under the hood.
- **3.2 Spectral Methods: Eigenbases, Filtering and Smoothing.** Builds the eigenbasis view and uses it to design filters and smoothing operators.
- **3.3 First Look at Variational Inference: The ELBO as a KL Objective.** Presents the ELBO as a KL objective and reads the variational gap as a fit-versus-prior trade.
- **3.4 Checkpoint: Precision, Spectra and Variational Bounds.** Closes the course by testing whether you can connect numerical precision, spectral structure and the variational bound into one picture.

## How the course works

The course runs about **320 minutes across 12 lessons**, roughly 25 to 30 minutes per lesson. Each module has three theory lessons and one checkpoint.

- **Theory lessons** carry the core material as short explanations, visuals and worked examples in depth. The examples are the practice: you are meant to follow the arithmetic, predict the next step, and check yourself against the derivation.
- **Checkpoint lessons** are quizzes that test judgment, not recall. They ask you to choose an objective, a factorization or a regularization strength and justify it.
- **No labs or projects.** This is a deep-dive reading course. The work happens in the examples and in your own reasoning.

You can take it linearly, and for a first pass you should: Module 1 sets up the objectives, Module 2 sets up the linear algebra machinery, and Module 3 combines them with numerical precision and the variational bound. The modules are loosely coupled, so if you already know the information-theoretic objectives cold, you can start at Module 2 and return to 1.3 for the forward-versus-reverse KL contrast, which the later lessons lean on. Do not skip 3.1: the floating-point material is the spine of the course, and the later examples assume it.

## Contents

- **🧮 Information-Theoretic Training Objectives**
  - [📉 Entropy, Cross-Entropy and KL: One Family of Losses](https://everwise.study/courses/mathematics-for-machine-learning-5-information-theory-matrix-methods-and/entropy-cross-entropy-kl-one-family.md)
  - [🔗 Mutual Information as an Objective: InfoNCE and Contrastive Bounds](https://everwise.study/courses/mathematics-for-machine-learning-5-information-theory-matrix-methods-and/mutual-information-objective-infonce-contrastive-bounds.md) (sign-in required)
  - [🪞 Forward vs Reverse KL: Mode Covering, Mode Seeking and Failure Modes](https://everwise.study/courses/mathematics-for-machine-learning-5-information-theory-matrix-methods-and/forward-vs-reverse-kl-mode-covering-seeking.md)
  - [✅ Checkpoint: Information-Theoretic Objectives](https://everwise.study/courses/mathematics-for-machine-learning-5-information-theory-matrix-methods-and/checkpoint-information-theoretic-objectives.md)
- **🔢 Solving and Regularizing Linear Systems**
  - [🧱 QR and Cholesky: Factorizations That Solve](https://everwise.study/courses/mathematics-for-machine-learning-5-information-theory-matrix-methods-and/qr-cholesky-factorizations-that-solve.md)
  - [🔍 Full SVD: Pseudoinverse, Truncation and Rank Control](https://everwise.study/courses/mathematics-for-machine-learning-5-information-theory-matrix-methods-and/full-svd-pseudoinverse-truncation-rank-control.md)
  - [⚖️ Conditioning, Ill-Posed Problems and Tikhonov Regularization](https://everwise.study/courses/mathematics-for-machine-learning-5-information-theory-matrix-methods-and/conditioning-ill-posed-tikhonov-regularization.md)
  - [✅ Checkpoint: Linear Systems in Practice](https://everwise.study/courses/mathematics-for-machine-learning-5-information-theory-matrix-methods-and/checkpoint-linear-systems-in-practice.md)
- **🌊 Floating Point, Spectral Methods and Variational Inference**
  - [🧊 Floating Point, Catastrophic Cancellation and Mixed Precision](https://everwise.study/courses/mathematics-for-machine-learning-5-information-theory-matrix-methods-and/floating-point-catastrophic-cancellation-mixed-precision.md)
  - [🌈 Spectral Methods: Eigenbases, Filtering and Smoothing](https://everwise.study/courses/mathematics-for-machine-learning-5-information-theory-matrix-methods-and/spectral-methods-eigenbasis-filtering-smoothing.md)
  - [🎯 First Look at Variational Inference: The ELBO as a KL Objective](https://everwise.study/courses/mathematics-for-machine-learning-5-information-theory-matrix-methods-and/first-look-variational-inference-elbo-kl.md) (premium content)
  - [✅ Checkpoint: Precision, Spectra and Variational Bounds](https://everwise.study/courses/mathematics-for-machine-learning-5-information-theory-matrix-methods-and/checkpoint-precision-spectra-variational-bounds.md) (premium content)
- [Course cheatsheet](https://everwise.study/courses/mathematics-for-machine-learning-5-information-theory-matrix-methods-and/cheatsheet.md) (premium content)