๐ŸŽš๏ธ Logistic Regression to Softmax: Cross-Entropy Done Right

By Better Wise

Your finite-difference harness from the gradient-checking lesson is the acceptance test for everything below. No gradient enters the capstone repo until the checker agrees to a relative error around 1e-7. In this lesson you derive the logistic and softmax gradients on paper first, then write fifteen lines of NumPy that survive the checker.

By the end you should be able to:

  • Derive $\frac{\partial L}{\partial z} = p - y$ for logistic regression from the chain rule, on paper.
  • Compute cross-entropy from logits with log-sum-exp, and explain why np.log(sigmoid(z)) dies at large $|z|$.
  • Derive the softmax Jacobian and show why it collapses against cross-entropy.
  • State every tensor shape in a vectorized train_step from memory.

Sigmoid, Log-Odds, and What Cross-Entropy Penalizes

The logistic model is a linear model in log-odds:

$$ z = w^\top x + b, \qquad p = \sigma(z) = \frac{1}{1 + e^{-z}}, \qquad \log\frac{p}{1-p} = z $$

Cross-entropy is the negative Bernoulli log-likelihood:

$$ L = -\left[y \log p + (1-y)\log(1-p)\right] $$

Cross-entropy is what makes the sigmoid gradient collapse to a subtraction, and squared error is what prevents it. If you use $\tfrac12(p-y)^2$, the chain rule drags in $\sigma'(z) = p(1-p) \le 0.25$, so a confidently wrong prediction ($p = 0.99$, $y = 0$) produces a gradient of roughly $0.0001$. The model learns slowest exactly where it is most wrong.

Exercise 1. Derive the squared-error gradient $w$ for a single sigmoid output, then explain in one sentence why cross-entropy avoids the problem. Answer

$\frac{\partial L}{\partial z} = (p-y),p(1-p)$, so $\frac{\partial L}{\partial w} = (p-y)p(1-p)x$. The extra $p(1-p)$ factor shrinks to zero at saturation. Cross-entropy gives $\frac{\partial L}{\partial z} = p-y$, which stays large (magnitude up to 1) no matter how saturated the sigmoid is.

A Log-Loss That Survives $|z| = 50$

Simplifying $-\log\sigma(z)$ gives $L = \log(1+e^z) - yz$. The naive version overflows: $e^{50}$ is already $5\times10^{21}$ and $e^{1000}$ is inf. Use the softplus identity:

$$ L = \max(z,0) + \log!\left(1 + e^{-|z|}\right) - yz $$

In NumPy this is one call: np.logaddexp(0, z) - y * z. The inner exponential is always $\le 1$, so nothing overflows and nothing underflows to $\log(0)$. _____ is the whole stable loss. Do not clip probabilities as a first resort; clip only when a library API forces you to.

Chain Rule on Paper: $\partial L / \partial z = p - y$

Two pieces, multiplied:

  • $\frac{\partial L}{\partial p} = -\frac{y}{p} + \frac{1-y}{1-p} = \frac{p-y}{p(1-p)}$
  • $\frac{\partial p}{\partial z} = p(1-p)$, the sigmoid derivative

The $p(1-p)$ cancels exactly: $\frac{\partial L}{\partial z} = p - y$. The sigmoid derivative is _____. For a batch of $N$ examples with $X \in \mathbb{R}^{N\times D}$:

p = 1.0 / (1.0 + np.exp(-(X @ w + b)))   # (N,)
dZ = (p - y) / N                          # (N,)
dW = X.T @ dZ                             # (D,)
db = dZ.sum()                             # scalar

Exercise 2. Your checker reports a 1e-2 relative error on dW but 1e-9 on db. Where is the bug? Answer

Almost certainly shape broadcasting: a (N,1) column vector of residuals multiplied against X gives an (N,N) matrix silently. Print dZ.shape first. This is why the checker is run before the optimizer ever touches the weights.

Softmax: $K$ Scores, One Distribution

For $K$ classes, keep one weight vector per class, so $W \in \mathbb{R}^{K\times D}$ and $Z = XW^\top + b$ is $N\times K$:

$$ p_k = \frac{e^{z_k}}{\sum_j e^{z_j}} $$

Softmax is shift-invariant: adding any constant to all logits leaves the probabilities unchanged. That is both a numerical gift and a modeling fact, since the _____ is subtracted before exponentiating, and only $K-1$ directions in logit space are identifiable.

Never compute np.log(softmax(Z)). Track logits instead:

def log_softmax(Z):
    Zs = Z - Z.max(axis=1, keepdims=True)       # overflow guard
    return Zs - np.log(np.exp(Zs).sum(axis=1, keepdims=True))

The Softmax Jacobian Collapses Against Cross-Entropy

The Jacobian is $\frac{\partial p_k}{\partial z_j} = p_k(\delta_{kj} - p_j)$: _____. That matrix is $K\times K$ and you should never build it. With $L = -\sum_k y_k \log p_k$:

$$ \frac{\partial L}{\partial z_j} = \sum_k \frac{-y_k}{p_k}, p_k(\delta_{kj} - p_j) = -y_j + p_j \sum_k y_k = p_j - y_j $$

The same subtraction as logistic regression. One-hot labels make $\sum_k y_k = 1$, which is the line where the Jacobian vanishes.

def train_step(W, b, X, Y):          # Y: (N, K) one-hot
    Z = X @ W.T + b                   # (N, K)
    logP = log_softmax(Z)             # (N, K)
    loss = -np.sum(Y * logP) / X.shape[0]
    dZ = (np.exp(logP) - Y) / X.shape[0]   # (N, K)
    return loss, dZ.T @ X, dZ.sum(axis=0)  # (K, D), (K,)

Note that np.exp(logP) is just the softmax, so the gradient never needs the raw logits twice.

Exercise 3. With $K=2$, the softmax model has twice the parameters of logistic regression. Verify they express the same function family, and name the redundancy. Answer

$p_1 = \sigma(z_1 - z_2)$ by dividing numerator and denominator by $e^{z_1}$, so only the difference $w_1 - w_2$ matters. The redundancy is shift invariance: adding a constant vector to both weight rows and both biases leaves predictions unchanged. L2 regularization or pinning one row to zero removes it.

Field Notes: Where Stable Implementations Actually Break

  • Check dZ before dW. Every chain-rule bug I have debugged in student code lived in the residual, not in the matrix product.
  • Standardize features. Softmax with unstandardized inputs saturates all rows and the loss plateaus at $\log K$.
  • Divide by $N$ once. Either in dZ or at the end, never both.
  • Put regularization inside the loss function you gradient-check. Adding $\lambda W$ in the optimizer while checking the unregularized loss is a false alarm you will waste an hour on.
  • Labels as one-hot $Y$ of shape (N, K) make the gradient a one-liner. Integer labels require fancy indexing and are the usual source of a transposed dW.

What This Unlocks in the Capstone

You now have two models whose gradients you derived, not copied: logistic regression as the $K=2$ special case, softmax with $\frac{\partial L}{\partial z} = p - y$, and a log-loss computed from logits that never overflows. Next comes the Module 1 checkpoint, where you will implement both against a fixed random seed and defend the gradient check numbers before moving to unsupervised learning.

Front: What does the logistic model assume is linear in x? Back: The log-odds: log(p/(1-p)) = w^T x + b.

Front: Why is squared error a poor loss for a sigmoid output? Back: Its gradient carries the factor p(1-p) <= 0.25, so it stalls when the model is confidently wrong.

Front: Numerically stable logistic loss in one expression? Back: np.logaddexp(0, z) - y * z, the softplus form of the Bernoulli NLL.

Front: Derivative of the sigmoid with respect to z? Back: p(1 - p).

Front: Derivative of logistic cross-entropy with respect to the logit z? Back: p - y, because p(1-p) cancels.

Front: Batch gradient for logistic regression? Back: dW = X^T (p - y) / N, db = mean(p - y).

Front: Why subtract the row max before exponentiating in softmax? Back: Softmax is shift-invariant, so it prevents overflow without changing probabilities.

Front: How do you compute log-softmax stably? Back: Zs = Z - max(Z); log_softmax = Zs - log(sum(exp(Zs))).

Front: Cross-entropy of a softmax model directly from logits? Back: -z_y + logsumexp(z), no probabilities needed.

Front: Softmax Jacobian entry dp_k/dz_j? Back: p_k(delta_kj - p_j).

Front: Softmax plus cross-entropy gradient with respect to logits? Back: dZ = (P - Y) / N, the same p - y structure as logistic regression.

Front: dW for softmax with W of shape (K, D) and Z = X W^T + b? Back: dW = dZ^T X, shape (K, D); db = dZ.sum(axis=0).

Front: How many effective logit directions does a K-class softmax have? Back: K - 1, due to shift invariance; the K-th is redundant.

Logistic regression models the _____ as a linear function of the features. The stable form of the logistic loss is np.logaddexp(0, z) - y * z, and its derivative with respect to z is _____. Softmax subtracts the _____ before exponentiating because the transform is shift-invariant. The softmax Jacobian is dp_k/dz_j = _____, which collapses to p - y when composed with cross-entropy. For a batch, the softmax weight gradient is dW = _____.