# ✅ Checkpoint: PCA, k-means and GMM By Better Wise ## Quiz 1. You fit PCA on a data matrix X with shape (500, 30) without subtracting the column means. What is the most likely effect on the first principal component and the spectrum? - All singular values become equal, so the explained variance ratios are uniform. - The first component becomes exactly the all-ones vector regardless of the data. - Nothing changes, because SVD is invariant to translation of the rows. - The first right singular vector tilts toward the mean vector and s[0] is inflated by the between-mean energy. 2. A PCA fit returns singular values s = [10, 4, 2, 1, 1] with n = 101 samples. What is the cumulative explained variance ratio at k = 2? - About 0.93 - About 0.80 - About 0.95 - About 0.50 3. Which statements about PCA via SVD are correct? Select all that apply. - The variance along component i is s_i^2 / (n - 1). - The rows of Vt hold the principal directions, and the scores are U scaled columnwise by s. - The squared Frobenius reconstruction error of a rank-k truncation equals the sum of the squared discarded singular values. - Two PCA runs on the same data can return components with flipped signs, so compare fits by absolute cosine similarity or apply svd_flip. - The Kaiser rule (eigenvalue > 1) is valid on the raw covariance matrix without standardization. 4. In k-means++, after the first center is chosen uniformly at random, each subsequent center is drawn with probability proportional to the ____ to the nearest already chosen center. 5. Which statements about Lloyd's algorithm for k-means are correct? Select all that apply. - The assignment step assigns each point to its nearest centroid and is the exact minimizer of inertia for fixed centroids. - You should select k by choosing the value that minimizes inertia. - The update step sets each centroid to the mean of its assigned points, the exact minimizer for a fixed assignment. - Inertia decreases monotonically, so the algorithm reaches a fixed point, but the result is a local minimum that depends on initialization. - An empty cluster should be handled by re-seeding that centroid, typically at the point farthest from its own centroid. 6. Your Lloyd loop converges quickly with k = 5, but inertia varies by 40 percent across random seeds. What is the correct reporting practice? - Increase the iteration count until all seeds agree exactly. - Switch to selecting k by minimum inertia to remove the variance. - Report the best run as if it were typical, since it shows what the algorithm can achieve. - Report the distribution of inertia across restarts (for example best, median, worst) and use k-means++ with a fixed restart budget. 7. In the GMM M-step, the mixing weight is updated as pi_k = ____, where N_k is the total responsibility assigned to component k and N is the number of points. 8. Why must the E-step of a GMM compute responsibilities using log-sum-exp rather than exponentiating the Gaussian densities directly? - Because log-sum-exp removes the need to regularize the covariance matrices. - Because log-sum-exp makes the responsibilities exactly 0 or 1, which speeds up the M-step. - Because the covariance matrix is not symmetric and cannot be inverted otherwise. - Because the Gaussian log density can be very negative, so exponentiating first underflows to 0 and every responsibility becomes nan. 9. Which statements about the EM algorithm for Gaussian mixtures are correct? Select all that apply. - When responsibilities become hard zeros and ones, the GMM reduces exactly to k-means with spherical, equal covariances. - The M-step covariance uses denominator N_k, not N and not N_k - 1, because the MLE does not correct for degrees of freedom. - The E-step makes the ELBO tight at the posterior, and the M-step maximizes it, so the observed-data log-likelihood is non-decreasing. - Adding reg * I to every covariance keeps the likelihood finite when a component collapses onto few points. - A sudden log-likelihood spike of hundreds of nats is a healthy sign that EM found a better global optimum. 10. Write a short design note (about 200 words) explaining how you would chain PCA, k-means, and a GMM in your project, and how you would defend each choice with numbers. Include the preprocessing, the component count, the restart budget, and the convergence criterion. --- ## Contents Part of [🧮 Mathematics for Machine Learning 4: Build Core ML Algorithms from Scratch](https://everwise.study/courses/mathematics-for-machine-learning-4-build-core-ml-algorithms-from-scratch.md) - **📈 Linear and Cross-Entropy Models from Scratch** - [📉 Linear Regression with Normal Equations and Gradient Descent](https://everwise.study/courses/mathematics-for-machine-learning-4-build-core-ml-algorithms-from-scratch/linear-regression-normal-equations-and-gradient-descent.md) - [🔍 Gradient Checking with a Finite-Difference Harness](https://everwise.study/courses/mathematics-for-machine-learning-4-build-core-ml-algorithms-from-scratch/gradient-checking-finite-difference-harness.md) (sign-in required) - [🎚️ Logistic Regression to Softmax: Cross-Entropy Done Right](https://everwise.study/courses/mathematics-for-machine-learning-4-build-core-ml-algorithms-from-scratch/logistic-regression-to-softmax-cross-entropy.md) - [✅ Checkpoint: Linear and Cross-Entropy Models](https://everwise.study/courses/mathematics-for-machine-learning-4-build-core-ml-algorithms-from-scratch/checkpoint-linear-and-cross-entropy-models.md) - **🔵 Unsupervised Learning from Scratch** - [🧭 PCA via SVD: Explained Variance and Component Selection](https://everwise.study/courses/mathematics-for-machine-learning-4-build-core-ml-algorithms-from-scratch/pca-via-svd-explained-variance.md) - [🌀 k-means: Lloyd's Algorithm and k-means++ Initialization](https://everwise.study/courses/mathematics-for-machine-learning-4-build-core-ml-algorithms-from-scratch/k-means-lloyd-and-k-means-plus-plus.md) - [🔔 Gaussian Mixture Models with Expectation Maximization](https://everwise.study/courses/mathematics-for-machine-learning-4-build-core-ml-algorithms-from-scratch/gaussian-mixture-models-expectation-maximization.md) - [✅ Checkpoint: PCA, k-means and GMM](https://everwise.study/courses/mathematics-for-machine-learning-4-build-core-ml-algorithms-from-scratch/checkpoint-pca-kmeans-and-gmm.md) — this lesson - **🧠 Backpropagation and Benchmark Capstone** - [🔗 Backpropagation for a One-Hidden-Layer MLP](https://everwise.study/courses/mathematics-for-machine-learning-4-build-core-ml-algorithms-from-scratch/backpropagation-for-one-hidden-layer-mlp.md) - [📊 Cross-Validation and a Fair Baseline Comparison Protocol](https://everwise.study/courses/mathematics-for-machine-learning-4-build-core-ml-algorithms-from-scratch/cross-validation-and-baseline-comparison-protocol.md) - [🏁 Capstone: NumPy Implementations vs Library Baselines](https://everwise.study/courses/mathematics-for-machine-learning-4-build-core-ml-algorithms-from-scratch/capstone-numpy-vs-library-baselines.md) (premium content) - [✅ Checkpoint: Backpropagation and Model Comparison](https://everwise.study/courses/mathematics-for-machine-learning-4-build-core-ml-algorithms-from-scratch/checkpoint-backpropagation-and-model-comparison.md) (premium content) - [Course cheatsheet](https://everwise.study/courses/mathematics-for-machine-learning-4-build-core-ml-algorithms-from-scratch/cheatsheet.md) (premium content)