# 🎚️ Logistic Regression to Softmax: Cross-Entropy Done Right

By Better Wise

Your finite-difference harness from the gradient-checking lesson is the acceptance test for everything below. No gradient enters the capstone repo until the checker agrees to a relative error around 1e-7. In this lesson you derive the logistic and softmax gradients on paper first, then write fifteen lines of NumPy that survive the checker.

By the end you should be able to:

- Derive $\frac{\partial L}{\partial z} = p - y$ for logistic regression from the chain rule, on paper.
- Compute cross-entropy from logits with log-sum-exp, and explain why `np.log(sigmoid(z))` dies at large $|z|$.
- Derive the softmax Jacobian and show why it collapses against cross-entropy.
- State every tensor shape in a vectorized `train_step` from memory.

## Sigmoid, Log-Odds, and What Cross-Entropy Penalizes

The logistic model is a linear model in log-odds:

$$
z = w^\top x + b, \qquad p = \sigma(z) = \frac{1}{1 + e^{-z}}, \qquad \log\frac{p}{1-p} = z
$$

Cross-entropy is the negative Bernoulli log-likelihood:

$$
L = -\left[y \log p + (1-y)\log(1-p)\right]
$$

**Cross-entropy is what makes the sigmoid gradient collapse to a subtraction**, and squared error is what prevents it. If you use $\tfrac12(p-y)^2$, the chain rule drags in $\sigma'(z) = p(1-p) \le 0.25$, so a confidently wrong prediction ($p = 0.99$, $y = 0$) produces a gradient of roughly $0.0001$. The model learns slowest exactly where it is most wrong.

**Exercise 1.** Derive the squared-error gradient $w$ for a single sigmoid output, then explain in one sentence why cross-entropy avoids the problem.
**Answer**

$\frac{\partial L}{\partial z} = (p-y)\,p(1-p)$, so $\frac{\partial L}{\partial w} = (p-y)p(1-p)x$. The extra $p(1-p)$ factor shrinks to zero at saturation. Cross-entropy gives $\frac{\partial L}{\partial z} = p-y$, which stays large (magnitude up to 1) no matter how saturated the sigmoid is.

## A Log-Loss That Survives $|z| = 50$

Simplifying $-\log\sigma(z)$ gives $L = \log(1+e^z) - yz$. The naive version overflows: $e^{50}$ is already $5\times10^{21}$ and $e^{1000}$ is `inf`. Use the softplus identity:

$$
L = \max(z,0) + \log\!\left(1 + e^{-|z|}\right) - yz
$$

In NumPy this is one call: `np.logaddexp(0, z) - y * z`. The inner exponential is always $\le 1$, so nothing overflows and nothing underflows to $\log(0)$. _____ is the whole stable loss. Do not clip probabilities as a first resort; clip only when a library API forces you to.

## Chain Rule on Paper: $\partial L / \partial z = p - y$

Two pieces, multiplied:

- $\frac{\partial L}{\partial p} = -\frac{y}{p} + \frac{1-y}{1-p} = \frac{p-y}{p(1-p)}$
- $\frac{\partial p}{\partial z} = p(1-p)$, the sigmoid derivative

The $p(1-p)$ cancels exactly: $\frac{\partial L}{\partial z} = p - y$. The sigmoid derivative is _____. For a batch of $N$ examples with $X \in \mathbb{R}^{N\times D}$:

```python
p = 1.0 / (1.0 + np.exp(-(X @ w + b)))   # (N,)
dZ = (p - y) / N                          # (N,)
dW = X.T @ dZ                             # (D,)
db = dZ.sum()                             # scalar
```

**Exercise 2.** Your checker reports a 1e-2 relative error on `dW` but 1e-9 on `db`. Where is the bug?
**Answer**

Almost certainly shape broadcasting: a `(N,1)` column vector of residuals multiplied against `X` gives an `(N,N)` matrix silently. Print `dZ.shape` first. This is why the checker is run before the optimizer ever touches the weights.

## Softmax: $K$ Scores, One Distribution

For $K$ classes, keep one weight vector per class, so $W \in \mathbb{R}^{K\times D}$ and $Z = XW^\top + b$ is $N\times K$:

$$
p_k = \frac{e^{z_k}}{\sum_j e^{z_j}}
$$

Softmax is shift-invariant: adding any constant to all logits leaves the probabilities unchanged. That is both a numerical gift and a modeling fact, since the _____ is subtracted before exponentiating, and only $K-1$ directions in logit space are identifiable.

Never compute `np.log(softmax(Z))`. Track logits instead:

```python
def log_softmax(Z):
    Zs = Z - Z.max(axis=1, keepdims=True)       # overflow guard
    return Zs - np.log(np.exp(Zs).sum(axis=1, keepdims=True))
```

## The Softmax Jacobian Collapses Against Cross-Entropy

The Jacobian is $\frac{\partial p_k}{\partial z_j} = p_k(\delta_{kj} - p_j)$: _____. That matrix is $K\times K$ and you should never build it. With $L = -\sum_k y_k \log p_k$:

$$
\frac{\partial L}{\partial z_j} = \sum_k \frac{-y_k}{p_k}\, p_k(\delta_{kj} - p_j) = -y_j + p_j \sum_k y_k = p_j - y_j
$$

The same subtraction as logistic regression. One-hot labels make $\sum_k y_k = 1$, which is the line where the Jacobian vanishes.

```python
def train_step(W, b, X, Y):          # Y: (N, K) one-hot
    Z = X @ W.T + b                   # (N, K)
    logP = log_softmax(Z)             # (N, K)
    loss = -np.sum(Y * logP) / X.shape[0]
    dZ = (np.exp(logP) - Y) / X.shape[0]   # (N, K)
    return loss, dZ.T @ X, dZ.sum(axis=0)  # (K, D), (K,)
```

Note that `np.exp(logP)` is just the softmax, so the gradient never needs the raw logits twice.

**Exercise 3.** With $K=2$, the softmax model has twice the parameters of logistic regression. Verify they express the same function family, and name the redundancy.
**Answer**

$p_1 = \sigma(z_1 - z_2)$ by dividing numerator and denominator by $e^{z_1}$, so only the difference $w_1 - w_2$ matters. The redundancy is shift invariance: adding a constant vector to both weight rows and both biases leaves predictions unchanged. L2 regularization or pinning one row to zero removes it.

## Field Notes: Where Stable Implementations Actually Break

- **Check `dZ` before `dW`.** Every chain-rule bug I have debugged in student code lived in the residual, not in the matrix product.
- **Standardize features.** Softmax with unstandardized inputs saturates all rows and the loss plateaus at $\log K$.
- **Divide by $N$ once.** Either in `dZ` or at the end, never both.
- **Put regularization inside the loss function** you gradient-check. Adding $\lambda W$ in the optimizer while checking the unregularized loss is a false alarm you will waste an hour on.
- **Labels as one-hot $Y$ of shape `(N, K)`** make the gradient a one-liner. Integer labels require fancy indexing and are the usual source of a transposed `dW`.

## What This Unlocks in the Capstone

You now have two models whose gradients you derived, not copied: logistic regression as the $K=2$ special case, softmax with $\frac{\partial L}{\partial z} = p - y$, and a log-loss computed from logits that never overflows. Next comes the Module 1 checkpoint, where you will implement both against a fixed random seed and defend the gradient check numbers before moving to unsupervised learning.

Front: What does the logistic model assume is linear in x?
Back: The log-odds: log(p/(1-p)) = w^T x + b.

Front: Why is squared error a poor loss for a sigmoid output?
Back: Its gradient carries the factor p(1-p) <= 0.25, so it stalls when the model is confidently wrong.

Front: Numerically stable logistic loss in one expression?
Back: np.logaddexp(0, z) - y * z, the softplus form of the Bernoulli NLL.

Front: Derivative of the sigmoid with respect to z?
Back: p(1 - p).

Front: Derivative of logistic cross-entropy with respect to the logit z?
Back: p - y, because p(1-p) cancels.

Front: Batch gradient for logistic regression?
Back: dW = X^T (p - y) / N, db = mean(p - y).

Front: Why subtract the row max before exponentiating in softmax?
Back: Softmax is shift-invariant, so it prevents overflow without changing probabilities.

Front: How do you compute log-softmax stably?
Back: Zs = Z - max(Z); log_softmax = Zs - log(sum(exp(Zs))).

Front: Cross-entropy of a softmax model directly from logits?
Back: -z_y + logsumexp(z), no probabilities needed.

Front: Softmax Jacobian entry dp_k/dz_j?
Back: p_k(delta_kj - p_j).

Front: Softmax plus cross-entropy gradient with respect to logits?
Back: dZ = (P - Y) / N, the same p - y structure as logistic regression.

Front: dW for softmax with W of shape (K, D) and Z = X W^T + b?
Back: dW = dZ^T X, shape (K, D); db = dZ.sum(axis=0).

Front: How many effective logit directions does a K-class softmax have?
Back: K - 1, due to shift invariance; the K-th is redundant.

Logistic regression models the _____ as a linear function of the features.
The stable form of the logistic loss is np.logaddexp(0, z) - y * z, and its derivative with respect to z is _____.
Softmax subtracts the _____ before exponentiating because the transform is shift-invariant.
The softmax Jacobian is dp_k/dz_j = _____, which collapses to p - y when composed with cross-entropy.
For a batch, the softmax weight gradient is dW = _____.

---

## Contents

Part of [🧮 Mathematics for Machine Learning 4: Build Core ML Algorithms from Scratch](https://everwise.study/courses/mathematics-for-machine-learning-4-build-core-ml-algorithms-from-scratch.md)

- **📈 Linear and Cross-Entropy Models from Scratch**
  - [📉 Linear Regression with Normal Equations and Gradient Descent](https://everwise.study/courses/mathematics-for-machine-learning-4-build-core-ml-algorithms-from-scratch/linear-regression-normal-equations-and-gradient-descent.md)
  - [🔍 Gradient Checking with a Finite-Difference Harness](https://everwise.study/courses/mathematics-for-machine-learning-4-build-core-ml-algorithms-from-scratch/gradient-checking-finite-difference-harness.md) (sign-in required)
  - [🎚️ Logistic Regression to Softmax: Cross-Entropy Done Right](https://everwise.study/courses/mathematics-for-machine-learning-4-build-core-ml-algorithms-from-scratch/logistic-regression-to-softmax-cross-entropy.md) — this lesson
  - [✅ Checkpoint: Linear and Cross-Entropy Models](https://everwise.study/courses/mathematics-for-machine-learning-4-build-core-ml-algorithms-from-scratch/checkpoint-linear-and-cross-entropy-models.md)
- **🔵 Unsupervised Learning from Scratch**
  - [🧭 PCA via SVD: Explained Variance and Component Selection](https://everwise.study/courses/mathematics-for-machine-learning-4-build-core-ml-algorithms-from-scratch/pca-via-svd-explained-variance.md)
  - [🌀 k-means: Lloyd's Algorithm and k-means++ Initialization](https://everwise.study/courses/mathematics-for-machine-learning-4-build-core-ml-algorithms-from-scratch/k-means-lloyd-and-k-means-plus-plus.md)
  - [🔔 Gaussian Mixture Models with Expectation Maximization](https://everwise.study/courses/mathematics-for-machine-learning-4-build-core-ml-algorithms-from-scratch/gaussian-mixture-models-expectation-maximization.md)
  - [✅ Checkpoint: PCA, k-means and GMM](https://everwise.study/courses/mathematics-for-machine-learning-4-build-core-ml-algorithms-from-scratch/checkpoint-pca-kmeans-and-gmm.md)
- **🧠 Backpropagation and Benchmark Capstone**
  - [🔗 Backpropagation for a One-Hidden-Layer MLP](https://everwise.study/courses/mathematics-for-machine-learning-4-build-core-ml-algorithms-from-scratch/backpropagation-for-one-hidden-layer-mlp.md)
  - [📊 Cross-Validation and a Fair Baseline Comparison Protocol](https://everwise.study/courses/mathematics-for-machine-learning-4-build-core-ml-algorithms-from-scratch/cross-validation-and-baseline-comparison-protocol.md)
  - [🏁 Capstone: NumPy Implementations vs Library Baselines](https://everwise.study/courses/mathematics-for-machine-learning-4-build-core-ml-algorithms-from-scratch/capstone-numpy-vs-library-baselines.md) (premium content)
  - [✅ Checkpoint: Backpropagation and Model Comparison](https://everwise.study/courses/mathematics-for-machine-learning-4-build-core-ml-algorithms-from-scratch/checkpoint-backpropagation-and-model-comparison.md) (premium content)
- [Course cheatsheet](https://everwise.study/courses/mathematics-for-machine-learning-4-build-core-ml-algorithms-from-scratch/cheatsheet.md) (premium content)