# 🎯 First Look at Variational Inference: The ELBO as a KL Objective

By Better Wise

> This lesson is premium content. Sign in and purchase the course to view the full content.

## The Bound You Can Actually Compute

Spectral filtering showed you how to move energy between eigenmodes while keeping the operator well conditioned. Variational inference applies the same instinct to probability: you cannot afford the true posterior, so you project it onto a family you can afford, and the projection metric is the KL divergence you already used as a training loss in Module 1. This lesson derives the ELBO from that projection and shows what the two competing terms inside it are fighting over.

By the end you should be able to:

- Derive $\log p(x) = \mathrm{ELBO} + \mathrm{KL}(q(z)\,\|\,p(z|x))$ without looking it up
- Read the ELBO as reconstruction minus regularization, and as energy plus entropy
- Evaluate a Gaussian ELBO by hand and interpret each number
- Recognize where the closed-form Gaussian KL silently loses precision in float32

### A posterior you cannot integrate

Take a latent variable model $p(x, z) = p(z)\,p(x \mid z)$ with a nonlinear decoder. Bayes gives $p(z \mid x) = p(x,z)/p(x)$, and the denominator is the evidence

$$
p(x) = \int p(x, z)\, dz
$$

which has no closed form once the decoder is a neural network. So pick a tractable family $q(z;\phi)$, usually a diagonal Gaussian, and treat inference as optimization over $\phi$.

### One line of algebra, and the ELBO appears

Start from the definition of the reverse KL between $q$ and the posterior:

$$
\mathrm{KL}(q(z)\,\|\,p(z \mid x)) = \mathbb{E}_q\!\left[\log q(z) - \log p(z \mid x)\right]
$$

Substitute $\log p(z \mid x) = \log p(x,z) - \log p(x)$. The term $\log p(x)$ does not depend on $z$, so it leaves the expectation:

$$
\log p(x) = \underbrace{\mathbb{E}_q\!\left[\log p(x,z) - \log q(z)\right]}_{\mathrm{ELBO}} + \mathrm{KL}(q(z)\,\|\,p(z \mid x))
$$

## In this lesson

- Two readings: fit the data, pay the prior
- Worked example: a one-dimensional Gaussian ELBO
- Where the closed-form KL loses digits
- What to Carry Forward


---

## Contents

Part of [🎲 Mathematics for Machine Learning 5: Information Theory, Matrix Methods and Numerical Stability](https://everwise.study/courses/mathematics-for-machine-learning-5-information-theory-matrix-methods-and.md)

- **🧮 Information-Theoretic Training Objectives**
  - [📉 Entropy, Cross-Entropy and KL: One Family of Losses](https://everwise.study/courses/mathematics-for-machine-learning-5-information-theory-matrix-methods-and/entropy-cross-entropy-kl-one-family.md)
  - [🔗 Mutual Information as an Objective: InfoNCE and Contrastive Bounds](https://everwise.study/courses/mathematics-for-machine-learning-5-information-theory-matrix-methods-and/mutual-information-objective-infonce-contrastive-bounds.md) (sign-in required)
  - [🪞 Forward vs Reverse KL: Mode Covering, Mode Seeking and Failure Modes](https://everwise.study/courses/mathematics-for-machine-learning-5-information-theory-matrix-methods-and/forward-vs-reverse-kl-mode-covering-seeking.md)
  - [✅ Checkpoint: Information-Theoretic Objectives](https://everwise.study/courses/mathematics-for-machine-learning-5-information-theory-matrix-methods-and/checkpoint-information-theoretic-objectives.md)
- **🔢 Solving and Regularizing Linear Systems**
  - [🧱 QR and Cholesky: Factorizations That Solve](https://everwise.study/courses/mathematics-for-machine-learning-5-information-theory-matrix-methods-and/qr-cholesky-factorizations-that-solve.md)
  - [🔍 Full SVD: Pseudoinverse, Truncation and Rank Control](https://everwise.study/courses/mathematics-for-machine-learning-5-information-theory-matrix-methods-and/full-svd-pseudoinverse-truncation-rank-control.md)
  - [⚖️ Conditioning, Ill-Posed Problems and Tikhonov Regularization](https://everwise.study/courses/mathematics-for-machine-learning-5-information-theory-matrix-methods-and/conditioning-ill-posed-tikhonov-regularization.md)
  - [✅ Checkpoint: Linear Systems in Practice](https://everwise.study/courses/mathematics-for-machine-learning-5-information-theory-matrix-methods-and/checkpoint-linear-systems-in-practice.md)
- **🌊 Floating Point, Spectral Methods and Variational Inference**
  - [🧊 Floating Point, Catastrophic Cancellation and Mixed Precision](https://everwise.study/courses/mathematics-for-machine-learning-5-information-theory-matrix-methods-and/floating-point-catastrophic-cancellation-mixed-precision.md)
  - [🌈 Spectral Methods: Eigenbases, Filtering and Smoothing](https://everwise.study/courses/mathematics-for-machine-learning-5-information-theory-matrix-methods-and/spectral-methods-eigenbasis-filtering-smoothing.md)
  - [🎯 First Look at Variational Inference: The ELBO as a KL Objective](https://everwise.study/courses/mathematics-for-machine-learning-5-information-theory-matrix-methods-and/first-look-variational-inference-elbo-kl.md) (premium content) — this lesson
  - [✅ Checkpoint: Precision, Spectra and Variational Bounds](https://everwise.study/courses/mathematics-for-machine-learning-5-information-theory-matrix-methods-and/checkpoint-precision-spectra-variational-bounds.md) (premium content)
- [Course cheatsheet](https://everwise.study/courses/mathematics-for-machine-learning-5-information-theory-matrix-methods-and/cheatsheet.md) (premium content)