# 🎲 Cheatsheet: Mathematics for Machine Learning 5: Information Theory, Matrix Methods and Numerical Stability

By Better Wise

> This cheatsheet is premium content. Sign in and purchase the course to view the full content.

## Module 1: Information-Theoretic Training Objectives

### 1.1 Entropy, Cross-Entropy, KL: One Family of Losses

| Quantity | Formula | Meaning |
| --- | --- | --- |
| Entropy | $H(p) = -\sum_x p(x)\log p(x)$ | Average surprise of $p$ (uncertainty) |
| Cross-entropy | $H(p,q) = -\sum_x p(x)\log q(x)$ | Bits to code $p$ using $q$ |
| KL divergence | $D_{KL}(p\|q) = \sum_x p(x)\log\frac{p(x)}{q(x)}$ | Excess cost of $q$ over $p$ |

Key identities:

- $H(p,q) = H(p) + D_{KL}(p\|q)$
- $D_{KL}(p\|q) \geq 0$, with equality iff $p = q$
- $D_{KL}$ is **not symmetric**: $D_{KL}(p\|q) \neq D_{KL}(q\|p)$
- Minimizing cross-entropy over $q$ = minimizing KL (since $H(p)$ is constant)

> **Note:** 
> Units: nats when using $\ln$, bits when using $\log_2$. Divide nats by $\ln 2$ to convert to bits.

```python
# Cross-entropy from logits (numerically stable via log-softmax)
import numpy as np

def cross_entropy(logits, target):
    # logits: (N, C), target: (N,) int labels
    logits = logits - logits.max(axis=1, keepdims=True)  # stabilize
    log_probs = logits - np.log(np.exp(logits).sum(axis=1, keepdims=True))
    return -log_probs[np.arange(len(target)), target].mean()
```

### 1.2 Mutual Information: InfoNCE and Contrastive Bounds

| Concept | Expression |
| --- | --- |
| Mutual information | $I(X;Y) = D_{KL}(p(x,y)\|p(x)p(y))$ |
| Equivalent form | $I(X;Y) = H(X) - H(X\|Y)$ |
| InfoNCE loss | $\mathcal{L} = -\mathbb{E}\left[\log\frac{e^{f(x,y^+)/\tau}}{\sum_{j} e^{f(x,y_j)/\tau}}\right]$ |
| InfoNCE bound | $I(X;Y) \geq \log K - \mathcal{L}_{\text{InfoNCE}}$ |

- $\tau$ = temperature; smaller $\tau$ sharpens the softmax (harder negatives)
- $K$ = number of candidates (batch size / negatives + 1)
- InfoNCE is a **lower bound** on MI; tighter with larger $K$
- Larger $K$ → tighter bound but more compute and more false negatives

Decision table: choosing the contrastive objective