✅ Checkpoint: Information-Theoretic Objectives

By Better Wise

Quiz

  1. Which decomposition correctly relates cross-entropy, entropy and KL divergence?
    • H(p,q) = KL(p||q) - H(p)
    • KL(p||q) = H(p,q) + H(p)
    • H(p,q) = H(q) + KL(q||p)
    • H(p,q) = H(p) + KL(p||q)
  2. For softmax plus cross-entropy with logits z, what is the gradient of the loss with respect to z?
    • log q - log p
    • p - q, the target minus the prediction
    • q - p, the softmax output minus the target distribution
    • q(1-q) times the error, as in squared error through a softmax
  3. Which statements about InfoNCE are correct? (Select all that apply.)
    • A larger batch lowers the reported loss by itself, even with an unchanged critic.
    • The optimal critic is the log density ratio log[p(y|x)/p(y)] plus a constant.
    • A per-device softmax uses the global batch as the effective number of negatives.
    • The bound saturates at log N nats because the loss is a cross-entropy over N candidates and cannot be negative.
    • The certified mutual information is log(N-1) - L, so the loss alone is not comparable across batch sizes.
  4. Which statements about forward and reverse KL are correct? (Select all that apply.)
    • Forward KL gradients require samples from q_theta and a score-function estimator.
    • Forward KL is infinite when p has mass where q is exactly zero.
    • Reverse KL produces mode seeking and zero forcing on a multimodal target.
    • A KL value from the forward direction can be compared directly with one from the reverse direction.
    • Fitting a single Gaussian to 0.5 N(-3,1) + 0.5 N(3,1) with forward KL gives mu = 0 and sigma^2 = 10.
  5. For one-hot labels the target entropy is zero, so minimizing cross-entropy is exactly minimizing the ____ divergence to the empirical distribution.
  6. In InfoNCE with N total candidates, the mutual information is lower bounded by ____ minus the loss.
  7. Variational inference minimizes the ____ KL direction, KL(q||p), because the ELBO equals log p(x) minus that term.
  8. Explain why computing a small KL divergence as H(p,q) - H(p) is numerically dangerous, and describe a safer computation. Refer to the magnitudes involved in float32.
  9. A colleague fits a single Gaussian to a bimodal target with reverse KL and reports a tight, confident fit. Explain what happened, why the fit is not a bug, and when you would prefer forward KL instead.
  10. Your contrastive run reports L = 2.1 nats with 512 candidates and another reports L = 2.1 with 64. Compute the certified mutual information in each case and explain why the raw losses are not comparable.

Sign in to submit answers.