# ✅ Checkpoint: Information-Theoretic Objectives By Better Wise ## Quiz 1. Which decomposition correctly relates cross-entropy, entropy and KL divergence? - H(p,q) = KL(p||q) - H(p) - KL(p||q) = H(p,q) + H(p) - H(p,q) = H(q) + KL(q||p) - H(p,q) = H(p) + KL(p||q) 2. For softmax plus cross-entropy with logits z, what is the gradient of the loss with respect to z? - log q - log p - p - q, the target minus the prediction - q - p, the softmax output minus the target distribution - q(1-q) times the error, as in squared error through a softmax 3. Which statements about InfoNCE are correct? (Select all that apply.) - A larger batch lowers the reported loss by itself, even with an unchanged critic. - The optimal critic is the log density ratio log[p(y|x)/p(y)] plus a constant. - A per-device softmax uses the global batch as the effective number of negatives. - The bound saturates at log N nats because the loss is a cross-entropy over N candidates and cannot be negative. - The certified mutual information is log(N-1) - L, so the loss alone is not comparable across batch sizes. 4. Which statements about forward and reverse KL are correct? (Select all that apply.) - Forward KL gradients require samples from q_theta and a score-function estimator. - Forward KL is infinite when p has mass where q is exactly zero. - Reverse KL produces mode seeking and zero forcing on a multimodal target. - A KL value from the forward direction can be compared directly with one from the reverse direction. - Fitting a single Gaussian to 0.5 N(-3,1) + 0.5 N(3,1) with forward KL gives mu = 0 and sigma^2 = 10. 5. For one-hot labels the target entropy is zero, so minimizing cross-entropy is exactly minimizing the ____ divergence to the empirical distribution. 6. In InfoNCE with N total candidates, the mutual information is lower bounded by ____ minus the loss. 7. Variational inference minimizes the ____ KL direction, KL(q||p), because the ELBO equals log p(x) minus that term. 8. Explain why computing a small KL divergence as H(p,q) - H(p) is numerically dangerous, and describe a safer computation. Refer to the magnitudes involved in float32. 9. A colleague fits a single Gaussian to a bimodal target with reverse KL and reports a tight, confident fit. Explain what happened, why the fit is not a bug, and when you would prefer forward KL instead. 10. Your contrastive run reports L = 2.1 nats with 512 candidates and another reports L = 2.1 with 64. Compute the certified mutual information in each case and explain why the raw losses are not comparable. --- ## Contents Part of [🎲 Mathematics for Machine Learning 5: Information Theory, Matrix Methods and Numerical Stability](https://everwise.study/courses/mathematics-for-machine-learning-5-information-theory-matrix-methods-and.md) - **🧮 Information-Theoretic Training Objectives** - [📉 Entropy, Cross-Entropy and KL: One Family of Losses](https://everwise.study/courses/mathematics-for-machine-learning-5-information-theory-matrix-methods-and/entropy-cross-entropy-kl-one-family.md) - [🔗 Mutual Information as an Objective: InfoNCE and Contrastive Bounds](https://everwise.study/courses/mathematics-for-machine-learning-5-information-theory-matrix-methods-and/mutual-information-objective-infonce-contrastive-bounds.md) (sign-in required) - [🪞 Forward vs Reverse KL: Mode Covering, Mode Seeking and Failure Modes](https://everwise.study/courses/mathematics-for-machine-learning-5-information-theory-matrix-methods-and/forward-vs-reverse-kl-mode-covering-seeking.md) - [✅ Checkpoint: Information-Theoretic Objectives](https://everwise.study/courses/mathematics-for-machine-learning-5-information-theory-matrix-methods-and/checkpoint-information-theoretic-objectives.md) — this lesson - **🔢 Solving and Regularizing Linear Systems** - [🧱 QR and Cholesky: Factorizations That Solve](https://everwise.study/courses/mathematics-for-machine-learning-5-information-theory-matrix-methods-and/qr-cholesky-factorizations-that-solve.md) - [🔍 Full SVD: Pseudoinverse, Truncation and Rank Control](https://everwise.study/courses/mathematics-for-machine-learning-5-information-theory-matrix-methods-and/full-svd-pseudoinverse-truncation-rank-control.md) - [⚖️ Conditioning, Ill-Posed Problems and Tikhonov Regularization](https://everwise.study/courses/mathematics-for-machine-learning-5-information-theory-matrix-methods-and/conditioning-ill-posed-tikhonov-regularization.md) - [✅ Checkpoint: Linear Systems in Practice](https://everwise.study/courses/mathematics-for-machine-learning-5-information-theory-matrix-methods-and/checkpoint-linear-systems-in-practice.md) - **🌊 Floating Point, Spectral Methods and Variational Inference** - [🧊 Floating Point, Catastrophic Cancellation and Mixed Precision](https://everwise.study/courses/mathematics-for-machine-learning-5-information-theory-matrix-methods-and/floating-point-catastrophic-cancellation-mixed-precision.md) - [🌈 Spectral Methods: Eigenbases, Filtering and Smoothing](https://everwise.study/courses/mathematics-for-machine-learning-5-information-theory-matrix-methods-and/spectral-methods-eigenbasis-filtering-smoothing.md) - [🎯 First Look at Variational Inference: The ELBO as a KL Objective](https://everwise.study/courses/mathematics-for-machine-learning-5-information-theory-matrix-methods-and/first-look-variational-inference-elbo-kl.md) (premium content) - [✅ Checkpoint: Precision, Spectra and Variational Bounds](https://everwise.study/courses/mathematics-for-machine-learning-5-information-theory-matrix-methods-and/checkpoint-precision-spectra-variational-bounds.md) (premium content) - [Course cheatsheet](https://everwise.study/courses/mathematics-for-machine-learning-5-information-theory-matrix-methods-and/cheatsheet.md) (premium content)