First Look at Variational Inference: The ELBO as a KL Objective

By Better Wise

The Bound You Can Actually Compute

Spectral filtering showed you how to move energy between eigenmodes while keeping the operator well conditioned. Variational inference applies the same instinct to probability: you cannot afford the true posterior, so you project it onto a family you can afford, and the projection metric is the KL divergence you already used as a training loss in Module 1. This lesson derives the ELBO from that projection and shows what the two competing terms inside it are fighting over.

By the end you should be able to:

• Derive $\log p(x) = \mathrm{ELBO} + \mathrm{KL}(q(z),|,p(z|x))$ without looking it up
• Read the ELBO as reconstruction minus regularization, and as energy plus entropy
• Evaluate a Gaussian ELBO by hand and interpret each number
• Recognize where the closed-form Gaussian KL silently loses precision in float32

A posterior you cannot integrate

Take a latent variable model $p(x, z) = p(z),p(x \mid z)$ with a nonlinear decoder. Bayes gives $p(z \mid x) = p(x,z)/p(x)$, and the denominator is the evidence

$$ p(x) = \int p(x, z), dz $$

which has no closed form once the decoder is a neural network. So pick a tractable family $q(z;\phi)$, usually a diagonal Gaussian, and treat inference as optimization over $\phi$.

One line of algebra, and the ELBO appears

Start from the definition of the reverse KL between $q$ and the posterior:

$$ \mathrm{KL}(q(z),|,p(z \mid x)) = \mathbb{E}_q!\left[\log q(z) - \log p(z \mid x)\right] $$

Substitute $\log p(z \mid x) = \log p(x,z) - \log p(x)$. The term $\log p(x)$ does not depend on $z$, so it leaves the expectation:

$$ \log p(x) = \underbrace{\mathbb{E}q!\left[\log p(x,z) - \log q(z)\right]}{\mathrm{ELBO}} + \mathrm{KL}(q(z),|,p(z \mid x)) $$

This lesson is premium content. Sign in and purchase the course to view the full content. Sign in