✅ Checkpoint: PCA, k-means and GMM

By Better Wise

Quiz

  1. You fit PCA on a data matrix X with shape (500, 30) without subtracting the column means. What is the most likely effect on the first principal component and the spectrum?
    • All singular values become equal, so the explained variance ratios are uniform.
    • The first component becomes exactly the all-ones vector regardless of the data.
    • Nothing changes, because SVD is invariant to translation of the rows.
    • The first right singular vector tilts toward the mean vector and s[0] is inflated by the between-mean energy.
  2. A PCA fit returns singular values s = [10, 4, 2, 1, 1] with n = 101 samples. What is the cumulative explained variance ratio at k = 2?
    • About 0.93
    • About 0.80
    • About 0.95
    • About 0.50
  3. Which statements about PCA via SVD are correct? Select all that apply.
    • The variance along component i is s_i^2 / (n - 1).
    • The rows of Vt hold the principal directions, and the scores are U scaled columnwise by s.
    • The squared Frobenius reconstruction error of a rank-k truncation equals the sum of the squared discarded singular values.
    • Two PCA runs on the same data can return components with flipped signs, so compare fits by absolute cosine similarity or apply svd_flip.
    • The Kaiser rule (eigenvalue > 1) is valid on the raw covariance matrix without standardization.
  4. In k-means++, after the first center is chosen uniformly at random, each subsequent center is drawn with probability proportional to the ____ to the nearest already chosen center.
  5. Which statements about Lloyd's algorithm for k-means are correct? Select all that apply.
    • The assignment step assigns each point to its nearest centroid and is the exact minimizer of inertia for fixed centroids.
    • You should select k by choosing the value that minimizes inertia.
    • The update step sets each centroid to the mean of its assigned points, the exact minimizer for a fixed assignment.
    • Inertia decreases monotonically, so the algorithm reaches a fixed point, but the result is a local minimum that depends on initialization.
    • An empty cluster should be handled by re-seeding that centroid, typically at the point farthest from its own centroid.
  6. Your Lloyd loop converges quickly with k = 5, but inertia varies by 40 percent across random seeds. What is the correct reporting practice?
    • Increase the iteration count until all seeds agree exactly.
    • Switch to selecting k by minimum inertia to remove the variance.
    • Report the best run as if it were typical, since it shows what the algorithm can achieve.
    • Report the distribution of inertia across restarts (for example best, median, worst) and use k-means++ with a fixed restart budget.
  7. In the GMM M-step, the mixing weight is updated as pi_k = ____, where N_k is the total responsibility assigned to component k and N is the number of points.
  8. Why must the E-step of a GMM compute responsibilities using log-sum-exp rather than exponentiating the Gaussian densities directly?
    • Because log-sum-exp removes the need to regularize the covariance matrices.
    • Because log-sum-exp makes the responsibilities exactly 0 or 1, which speeds up the M-step.
    • Because the covariance matrix is not symmetric and cannot be inverted otherwise.
    • Because the Gaussian log density can be very negative, so exponentiating first underflows to 0 and every responsibility becomes nan.
  9. Which statements about the EM algorithm for Gaussian mixtures are correct? Select all that apply.
    • When responsibilities become hard zeros and ones, the GMM reduces exactly to k-means with spherical, equal covariances.
    • The M-step covariance uses denominator N_k, not N and not N_k - 1, because the MLE does not correct for degrees of freedom.
    • The E-step makes the ELBO tight at the posterior, and the M-step maximizes it, so the observed-data log-likelihood is non-decreasing.
    • Adding reg * I to every covariance keeps the likelihood finite when a component collapses onto few points.
    • A sudden log-likelihood spike of hundreds of nats is a healthy sign that EM found a better global optimum.
  10. Write a short design note (about 200 words) explaining how you would chain PCA, k-means, and a GMM in your project, and how you would defend each choice with numbers. Include the preprocessing, the component count, the restart budget, and the convergence criterion.

Sign in to submit answers.