Forward vs Reverse KL: Mode Covering, Mode Seeking and Failure Modes

By Better Wise

In the previous lesson, InfoNCE appeared as a bound on mutual information, and that bound is itself a KL divergence between a joint and a product of marginals. The direction of that KL was not a cosmetic choice. Here we make direction the main object of study, because the direction decides whether a fitted model spreads out over everything the data does, or picks one explanation and ignores the rest.

By the end of this lesson you should be able to:

• write both directions and name the objective that optimizes each one
• predict covering versus collapse straight from the direction
• compute the optimum for a bimodal target fitted by one Gaussian
• explain why forward KL gradients are cheap and reverse KL gradients are not
• recognize the support and floating-point failures that make either loss infinite or useless

The two directions and what they forbid

Fix a target $p$ (data, or a posterior) and a model family $q_\theta$. There are two divergences, and they are not the same object.

The mechanism is in the two integrals. $\mathrm{KL}(p|q)=\mathbb{E}_p[\log p-\log q]$ blows up when $p$ has density where $q$ has none, so $q$ must dominate the support of $p$: that is what forces a fit to keep every mode. $\mathrm{KL}(q|p)=\mathbb{E}_q[\log q-\log p]$ blows up when $q$ has density where $p$ has none, so $q$ must hide inside the support of $p$: that is what forces it to drop modes it cannot reach without crossing empty space.

Two identities turn this into practice. Minimizing $\mathrm{KL}(p_{\text{data}}|q_\theta)$ is minimizing cross-entropy up to a constant, so every maximum likelihood run is forward KL. And since $\mathrm{ELBO}=\log p(x)-\mathrm{KL}(q|p)$, maximizing the ELBO is minimizing reverse KL, so every standard variational fit is reverse KL. You will use the second identity in the variational inference lesson; keep it parked.

flowchart TD
    A["🎯 What must the fit do?"] --> B{"Is every mode of p required?"}
    B -->|Yes| C["Forward KL<br/>KL(p, q)<br/>mass covering"]
    B -->|No, one good mode| D["Reverse KL<br/>KL(q, p)<br/>mode seeking"]
    C --> E["MLE / cross-entropy<br/>low-variance gradient"]
    D --> F["Variational inference<br/>score-function gradient"]

Worked example: one Gaussian, two modes

Take $p = \tfrac12\mathcal{N}(-3,1) + \tfrac12\mathcal{N}(3,1)$ and let $q$ be a single Gaussian $\mathcal{N}(\mu,\sigma^2)$. This is the smallest example where the two directions visibly disagree.

Forward. For an exponential family $q$, minimizing $\mathrm{KL}(p|q)$ is moment matching: $\mu^*=\mathbb{E}_p[x]=0$ and $\sigma^{2*}=\mathrm{Var}_p[x]=9+1=10$. The value is $\mathbb{E}_p[\log p]-\mathbb{E}_p[\log q]$, where $H(p)\approx 2.11$ nats (about $1.42$ per component plus $0.69$ of mixing entropy), and $\mathbb{E}_p[\log q]=-\tfrac12\log(20\pi)-\tfrac{10}{20}\approx-2.57$, giving $\mathrm{KL}\approx 0.46$ nats. Read the fit: $\sigma\approx 3.16$ against a true mode width of $1$. The model is more than three times too wide and puts most of its mass at $x=0$, where $p(0)=\phi(3)\approx 0.0044$. Samples from this fit look like grey mush between the modes.

Reverse. Take $q=\mathcal{N}(-3,1)$. Over the region where $q$ has mass, $\log p \approx \log\tfrac12+\log\mathcal{N}(x;-3,1)$, so the log ratio is the constant $\log 2$ and $\mathrm{KL}(q|p)=\log 2\approx 0.69$ nats. Widening or narrowing $q$ costs more, and straddling both modes is expensive because $q$ would have to pay for mass in the valley. The optimum locks onto one mode and gets that mode's width right. The other mode is simply gone.

Note that $0.46$ and $0.69$ are not comparable numbers. They score different objectives, and the collapsed fit is "better" under its own divergence while being a worse description of $p$ globally.

A compact support example makes the asymmetry sharper. Let $p$ be uniform on $(0,1)$ and $q=\mathcal{N}(0.5,1)$. Then $\mathrm{KL}(p|q)$ is finite, while $\mathrm{KL}(q|p)=\infty$ because $q$ puts mass outside $(0,1)$ where $p=0$. Reverse KL does not tolerate a support mismatch: it either forces $q$ inside $p$ or diverges.

Why the gradients behave so differently

Forward KL differentiates to a plain cross-entropy gradient:

$$ \nabla_\theta \mathrm{KL}(p|q_\theta) = -\mathbb{E}p\left[\nabla\theta \log q_\theta(x)\right] $$

Samples come from the data, the estimator is low variance, and no extra machinery is needed. This is why likelihood training is the default.

Reverse KL keeps the score function:

$$ \nabla_\theta \mathrm{KL}(q_\theta|p) = \mathbb{E}{q\theta}\left[\nabla_\theta \log q_\theta ,(\log q_\theta - \log p)\right] $$

The term $\nabla_\theta\log q_\theta$ is a score function, so the estimator is high variance and $q_\theta$ must be reparametrizable (or handled with analytic entropy terms). Every difficulty people report with variational inference, from noisy gradients to the need for control variates, traces back to this line.

Failure modes worth memorizing

• Blur (forward KL): correct mode weights, no plausible individual samples. The fit averages across modes because covering is cheaper than missing.
• Collapse and overconfidence (reverse KL): one mode kept, the rest discarded, posterior variance underestimated, credible intervals too narrow.
• Infinite loss from support mismatch: a bounded family against an unbounded target, or the reverse. Practitioners add jitter and pick families with adequate support for exactly this reason.
• Cross-direction comparison: KL is not symmetric and not a metric. Never report one number as "better" than a number from the other direction.
• Numerical decay: evaluating $\log p$ for a mixture with a far component underflows, and $\log q-\log p$ with tiny $q$ loses precision through cancellation. Use log-sum-exp for the mixture density and work in log space throughout.

Warning: A reverse KL estimate computed with importance weights can return a finite number while the true divergence is infinite, because the samples never land in the region that matters. Check support analytically before trusting the number.

Check yourself

A target is a mixture of 8 well-separated modes and your fit is a single Gaussian trained with forward KL. Describe the fit and the loss behavior as you widen the family.

The fit centers near the global mean with variance equal to the mixture variance, so it is far too wide relative to any mode and spends most mass in empty regions. Widening beyond the moment-matching variance increases the loss, because the penalty for placing density where $p$ is small outweighs the benefit of covering the outer modes.

You halve the temperature of a reverse-KL variational fit and the ELBO improves while held-out likelihood degrades. Is that a bug?

No. Reverse KL is mode seeking, so a sharper $q$ can fit one region of the posterior better while ignoring others. The ELBO only bounds the marginal likelihood under the assumed family, so it can improve while the approximation becomes less representative.

Choosing a direction on purpose

• Decide what the model is for before you pick the loss. Sampling realism argues for reverse KL behavior, calibrated coverage and density estimation argue for forward KL.
• If you need both, mix the directions or use a symmetric or $\alpha$-divergence, and say so explicitly in the writeup.
• Sanity-check the support: plot or bound $q$'s support against $p$'s. One infinite divergence destroys a training run quietly.
• Remember $-\mathbb{E}_p[\log q]$ equals forward KL plus the entropy of $p$. When $p$ is fixed, you can drop the entropy term, but it is not zero, and forgetting it makes reported numbers meaningless.
• Compute every mixture log-density with log-sum-exp. In a bimodal fit separated by six standard deviations, the naive sum loses all information about the far component in float32.

What to carry forward

Direction is a design decision, not a detail. Forward KL covers and overdispersion follows; reverse KL concentrates and collapse follows; support mismatches turn either one infinite; and only the forward direction gives you a cheap, low-variance gradient. The checkpoint next pulls entropy, cross-entropy, KL and mutual information into one family and asks you to pick the right member for a stated objective, including the direction.

Front: Which KL direction does maximum likelihood minimize? Back: KL(p_data || q_theta), the forward direction, which equals cross-entropy up to a constant.

Front: Which KL direction does variational inference minimize? Back: KL(q || p), the reverse direction, because ELBO = log p(x) - KL(q || p).

Front: Behavior of forward KL on a multimodal target? Back: Mass covering and mean seeking: q spreads out until no mode of p is missed.

Front: Behavior of reverse KL on a multimodal target? Back: Mode seeking and zero forcing: q collapses onto one mode and drops the others.

Front: Forward KL optimum for p = 0.5 N(-3,1) + 0.5 N(3,1) with a single Gaussian q? Back: mu = 0 and sigma^2 = 10, the true mean and variance, so the fit is about three times too wide.

Front: Reverse KL fit of a single Gaussian to p = 0.5 N(-3,1) + 0.5 N(3,1)? Back: It collapses onto one mode: mu = +3 or -3 with sigma^2 = 1, a confidently wrong local fit.

Front: Which direction enforces zero forcing, and what does that mean? Back: Reverse KL: wherever p(x) is near zero, q(x) is forced to zero too, so q cannot hedge across regions p rejects.

Front: When is forward KL infinite? Back: When p has mass somewhere q assigns exactly zero, since the integrand p log(p/q) diverges there.

Front: When is reverse KL infinite? Back: When q has mass somewhere p assigns exactly zero, since the integrand q log(q/p) diverges there.

Front: Why is the gradient of forward KL cheaper to estimate than reverse KL? Back: It is an expectation under p_data, so samples come from the dataset; reverse KL needs samples from q, which changes every step.

Front: Which divergence direction matches the mean and variance of a target? Back: Forward KL with a Gaussian q: the optimum sets mu to the target mean and sigma^2 to the target variance (moment matching).

Front: Which divergence direction is the objective inside the ELBO? Back: Reverse KL, KL(q || p), because log p(x) minus the ELBO equals exactly that term.

Front: Signature symptom of a forward-KL fit with limited capacity? Back: Overdispersion: the fit is broader than the target and places mass in low-density gaps between modes.

Front: Signature symptom of a reverse-KL fit with limited capacity? Back: Missing modes: the fit locks onto one region and assigns negligible mass to the others.

Front: Where does the alpha-divergence family place forward and reverse KL? Back: At the two endpoints: alpha = 1 recovers forward KL (mass covering), alpha = 0 recovers reverse KL (zero forcing).

Front: How do you diagnose which direction produced a suspicious fit? Back: Check the tails and the gaps: excess mass in gaps means forward KL, near-zero mass on a real mode means reverse KL.

Front: Why does reverse KL tolerate a badly underestimated variance? Back: Because underestimating variance costs little under q's own expectation, while spilling mass onto low-probability regions of p is punished heavily.

Maximizing likelihood minimizes _____, the forward direction, which differs from cross-entropy only by a constant. Variational inference minimizes _____, the reverse direction. Forward KL forbids q from being zero where p has mass, which produces _____ behavior. Reverse KL forbids q from placing mass where p is near zero, which produces _____ behavior and zero forcing. If p has mass on a region where q assigns exactly zero, the forward KL is _____.

Two rules of thumb to keep beside the equations: forward KL is what your data gives you for free, reverse KL is what your tractable approximation gives you for free, and whichever you pick, the geometry of the fit follows whether you asked for it or not. The checkpoint next will test exactly this: choosing the divergence direction from a stated downstream cost, not from habit.