Cheatsheet: Mathematics for Machine Learning 5: Information Theory, Matrix Methods and Numerical Stability
Module 1: Information-Theoretic Training Objectives
1.1 Entropy, Cross-Entropy, KL: One Family of Losses
| Quantity | Formula | Meaning |
|---|---|---|
| Entropy | $H(p) = -\sum_x p(x)\log p(x)$ | Average surprise of $p$ (uncertainty) |
| Cross-entropy | $H(p,q) = -\sum_x p(x)\log q(x)$ | Bits to code $p$ using $q$ |
| KL divergence | $D_{KL}(p|q) = \sum_x p(x)\log\frac{p(x)}{q(x)}$ | Excess cost of $q$ over $p$ |
Key identities:
- $H(p,q) = H(p) + D_{KL}(p|q)$
- $D_{KL}(p|q) \geq 0$, with equality iff $p = q$
- $D_{KL}$ is not symmetric: $D_{KL}(p|q) \neq D_{KL}(q|p)$
- Minimizing cross-entropy over $q$ = minimizing KL (since $H(p)$ is constant)
Note: Units: nats when using $\ln$, bits when using $\log_2$. Divide nats by $\ln 2$ to convert to bits.
# Cross-entropy from logits (numerically stable via log-softmax)
import numpy as np
def cross_entropy(logits, target):
# logits: (N, C), target: (N,) int labels
logits = logits - logits.max(axis=1, keepdims=True) # stabilize
log_probs = logits - np.log(np.exp(logits).sum(axis=1, keepdims=True))
return -log_probs[np.arange(len(target)), target].mean()
1.2 Mutual Information: InfoNCE and Contrastive Bounds
| Concept | Expression |
|---|---|
| Mutual information | $I(X;Y) = D_{KL}(p(x,y)|p(x)p(y))$ |
| Equivalent form | $I(X;Y) = H(X) - H(X|Y)$ |
| InfoNCE loss | $\mathcal{L} = -\mathbb{E}\left[\log\frac{e^{f(x,y^+)/\tau}}{\sum_{j} e^{f(x,y_j)/\tau}}\right]$ |
| InfoNCE bound | $I(X;Y) \geq \log K - \mathcal{L}_{\text{InfoNCE}}$ |
- $\tau$ = temperature; smaller $\tau$ sharpens the softmax (harder negatives)
- $K$ = number of candidates (batch size / negatives + 1)
- InfoNCE is a lower bound on MI; tighter with larger $K$
- Larger $K$ → tighter bound but more compute and more false negatives
Decision table: choosing the contrastive objective
This cheatsheet is premium content. Sign in and purchase the course to view the full content. Sign in