Entropy, Cross-Entropy and KL: One Family of Losses

By Better Wise

Every classifier you have trained logged a number called loss, and that number was almost certainly a cross-entropy. This lesson takes that number apart. Cross-entropy is not a primitive: it is entropy plus a KL divergence, and knowing which of the two terms your gradients actually touch explains most of the training behaviour you will meet later in this module. You already have expectations and matrix algebra; the only new machinery is reading $\sum_x p(x)\log p(x)$ as an expectation.

By the end you should be able to:

• write entropy, cross-entropy and KL as expectations and convert between them;
• decompose $H(p,q) = H(p) + \mathrm{KL}(p|q)$ and say which term is a property of the data;
• derive $\partial L/\partial z = q - p$ for softmax plus cross-entropy and read off what it means;
• spot the places where this loss is computed in a numerically fragile way.

Entropy: the cost of the data, not the model

For a distribution $p$ over outcomes $x$,

$$ H(p) = -\sum_x p(x)\log p(x) = \mathbb{E}_{x\sim p}[-\log p(x)] $$

This is the average number of nats (natural log) or bits ($\log_2$) needed to encode a draw from $p$ with a code built for $p$. Units are a constant factor apart: $\ln 2 \approx 0.693$, so 1 bit is 0.693 nats and the argmin never changes.

Two anchors: a uniform distribution over $K$ outcomes has $H = \log K$ (for $K = 1000$, 6.908 nats or 9.966 bits), and a point mass has $H = 0$. Entropy is a function of $p$ only. It contains no parameters, so no gradient can reduce it.

Cross-entropy: what you pay for using the wrong code

$$ H(p,q) = -\sum_x p(x)\log q(x) = \mathbb{E}_{x\sim p}[-\log q(x)] $$

Model $q$, data $p$. Cross-entropy is linear in $p$, which is why the mean of per-sample losses over a batch is exactly the cross-entropy against the empirical distribution. It is also not symmetric: $H(p,q) \neq H(q,p)$ in general.

Worked example (three classes). Take target $p = (0.7, 0.2, 0.1)$ and model $q = (0.5, 0.3, 0.2)$:

Check the KL directly: $0.7\ln(0.7/0.5) + 0.2\ln(0.2/0.3) + 0.1\ln(0.1/0.2) = 0.2355 - 0.0811 - 0.0693 = 0.0851$. The identity holds to the precision of the printed digits.

KL divergence: the part your parameters can move

$$ \mathrm{KL}(p|q) = \sum_x p(x)\log\frac{p(x)}{q(x)} = H(p,q) - H(p) $$

Gibbs' inequality gives $\mathrm{KL}(p|q) \ge 0$ with equality iff $p = q$. The KL is the excess cost from encoding $p$-data with a $q$-code, and it is the only term in the decomposition that depends on the model. Minimizing cross-entropy over parameters is therefore identical to minimizing $\mathrm{KL}(p|q)$; the two objectives have the same minimizer and the same gradients.

flowchart TD
  CE["📉<br/>H(p,q): the loss you log"] --> EN["🔒<br/>H(p): target entropy, fixed"]
  CE --> KLD["🎯<br/>KL(p||q): the part gradients shrink"]
  EN --> NOTE["Only the KL term depends on the parameters"]
  KLD --> NOTE

The hard-label case. In ordinary classification $p$ is one-hot, so $H(p) = 0$ and cross-entropy equals KL. Supervised classification with hard labels is divergence minimization against the empirical distribution, nothing more.

Why softmax plus cross-entropy trains the way it does

Let logits be $z \in \mathbb{R}^K$, $q = \mathrm{softmax}(z)$, and the loss for correct class $y$ be $L = -\log q_y$. Written in the form implementations actually use:

$$ L = -z_y + \log\sum_k e^{z_k} $$

Differentiating gives the cleanest result in this course:

$$ \frac{\partial L}{\partial z} = q - p $$

For hard labels, $\partial L/\partial z = q - e_y$. Three consequences follow immediately.

1. The gradient is bounded: each component lies in $[-1, 1]$, so logit updates never explode and the effective step size is roughly the learning rate.
2. The gradient vanishes only when the prediction is right, not when it is confident. A confidently correct example contributes almost nothing; a confidently wrong one contributes a near-maximal push.
3. The gradient is the error, so tools that inspect gradients in logit space are inspecting a probability residual.

Worked example (saturation). Two logits, correct class 0.

• Confident and correct: $z = (2.0, -1.0)$, $q = (0.9526, 0.0474)$, $L = 0.0486$ nats, gradient $(-0.0474, 0.0474)$.
• Confident and wrong: $z = (-1.0, 2.0)$, $q = (0.0474, 0.9526)$, $L = 3.049$ nats, gradient $(-0.9526, 0.9526)$.

Compare a squared error through the same softmax at the second point: its gradient in logit space carries a factor $q(1-q) \approx 0.045$, roughly 20 times smaller. That factor is the classic vanishing-gradient story, and cross-entropy removes it by construction.

Soft targets change the gradient, not the form. With label smoothing, distillation or mixup, $p$ is not one-hot and the gradient is still $q - p$. A softer $p$ produces a softer push, and the loss floor becomes $H(p) > 0$. Concretely: $K = 10$, smoothing $\epsilon = 0.1$ gives a target with $p_y = 0.91$ and 0.01 elsewhere, so $H(p) \approx 0.50$ nats. Your training loss will plateau near 0.50 and that is correct behaviour, not a bug.

Where the arithmetic bites. Implementations compute log_softmax and then take the negative log-probability. Doing log(softmax(z)) instead underflows: with a logit gap above roughly 90, softmax returns exactly 0 in float32 and log returns -inf. The same fragility appears if you compute a small KL as $H(p,q) - H(p)$: two numbers near 1.0 subtracted in float32 (machine epsilon $\approx 1.2\times10^{-7}$) leave a difference of $10^{-4}$ with about $10^{-3}$ relative error, and a difference of $10^{-6}$ is pure noise. Compute $\sum_x p(\log p - \log q)$ directly, or subtract log-probabilities, when the divergence is small.

Numerical habits that keep this loss honest

• Never softmax then log. Use log_softmax plus negative log-likelihood (torch.nn.CrossEntropyLoss does this internally, on logits).
• Never estimate a tiny KL by differencing two entropies. The cancellation is catastrophic exactly where your regularizer or ELBO term lives.
• Remember $H(p)$ is a floor. Cross-entropy below the target entropy is impossible, so a plateau at $0.5$ with smoothing is a correct answer, not a stall.
• Report nats or bits explicitly. Mixing them silently rescales your learning rate by 1.44.
• Use $q - p$ as your mental model of the gradient: if a debugging plot of the residual is not shrinking, the loss is not shrinking.

What to carry into the next lesson

Cross-entropy is entropy plus a KL, the KL is the trainable half, and its gradient in logit space is the probability residual $q - p$. Keep the decomposition in hand: the next lesson replaces the target $p$ with a joint distribution and builds InfoNCE, where the same KL appears between a joint and a product of marginals, giving mutual information as an objective.

Front: Decompose cross-entropy into entropy and KL. Back: H(p,q) = H(p) + KL(p||q).

Front: Definition of entropy H(p). Back: H(p) = -sum_x p(x) log p(x) = E_p[-log p(x)].

Front: Definition of cross-entropy H(p,q). Back: H(p,q) = -sum_x p(x) log q(x) = E_p[-log q(x)].

Front: What does Gibbs' inequality say about KL? Back: KL(p||q) >= 0, with equality if and only if p = q.

Front: Why does cross-entropy equal KL for hard labels? Back: A one-hot target has H(p) = 0, so the fixed entropy term disappears.

Front: Gradient of cross-entropy with respect to the logits. Back: dL/dz = q - p, the softmax output minus the target distribution.

Front: Why is log(softmax(z)) dangerous? Back: Softmax underflows to 0 for large logit gaps, so the log returns -inf. Use log_softmax or logsumexp.

Front: Entropy of the uniform distribution over K outcomes. Back: log K, which is 6.908 nats (9.966 bits) for K = 1000.

Front: How do nats and bits relate? Back: They differ by a constant factor of ln 2, so the argmin of any of these losses is unchanged.

Front: Is cross-entropy symmetric in p and q? Back: No, H(p,q) != H(q,p) in general, and KL is asymmetric as well.

Front: Why does label smoothing create a loss floor? Back: The smoothed target is not one-hot, so its entropy H(p) is positive and cannot be removed by training.

Front: Why does cross-entropy avoid the saturation that squared error shows through a softmax? Back: Its logit-space gradient q - p stays near maximal for confidently wrong predictions, with no q(1-q) damping factor.

Front: Safest way to compute a very small KL divergence. Back: Sum p(log p - log q) directly, or subtract log-probabilities, instead of differencing two entropy estimates.

Fill in the blanks before moving on:

• The decomposition that organizes this whole lesson is $H(p,q) =$ _____.
• For one-hot labels the entropy of the target is _____, so minimizing cross-entropy is exactly minimizing _____ to the empirical distribution.
• The gradient of cross-entropy with respect to the logits is _____, which is bounded in every component.
• Entropy is measured in _____ when the logarithm is natural.
• KL divergence is zero if and only if _____.