# ✅ Checkpoint: Linear and Cross-Entropy Models By Better Wise ## Quiz 1. You have a full-rank regression problem with n = 500 rows and d = 4 columns (including the intercept). Which solver should you use, and why? - Batch gradient descent, because the normal equations are always numerically unstable. - Batch gradient descent, because O(d^3) factorization is prohibitive at d = 4. - Neither: you must invert X^T X explicitly to get the answer. - The normal equations via np.linalg.lstsq, because O(nd^2 + d^3) is trivial at d = 4 and it gives the exact optimum in one shot. 2. Which statements about batch gradient descent for linear regression are correct? Select all that apply. - Features should be standardized to zero mean and unit variance so a single learning rate suits every coordinate. - If the loss oscillates or explodes, the learning rate is likely too large. - One iteration costs O(nd), one pass for the residual and one for the gradient. - You should monitor the parameter norm, not the loss, to decide convergence. - On full-rank data, GD should converge to a different optimum than the normal equations. 3. The gradient of the mean squared error J(w) = (1/2n)||Xw - y||^2 with respect to w is ____. 4. Your finite-difference check reports a worst relative error of about 0.96, and every coordinate fails by roughly the same factor. What is the most likely bug? - The check was run at a kink such as ReLU at 0. - An indexing or slicing error affecting a single coordinate. - The check was run in float32, so roundoff explains it. - A scale or normalization bug, such as a missing 1/n on the whole data term of the gradient. 5. Which practices make a finite-difference gradient check trustworthy? Select all that apply. - Check at all-zero parameters, since that is the most reproducible point. - Use the central difference (f(x + eps e_i) - f(x - eps e_i)) / (2 eps), whose truncation error is O(eps^2). - Compare the sums of the two gradients instead of the entries. - Compare gradients elementwise with the relative error |num - ana| / max(1e-8, |num| + |ana|). - Restore the perturbed entry after each coordinate so later evaluations are not corrupted. 6. For float64 arrays, the standard central-difference step size is about ____ (give the numeric value used in the lesson). 7. Why does cross-entropy, not squared error, give the clean gradient p - y for a sigmoid output? - The p(1-p) from the sigmoid derivative cancels exactly against the 1/(p(1-p)) factor in dL/dp. - Because squared error has no gradient at all for a sigmoid output. - Because cross-entropy removes the sigmoid from the model. - Because cross-entropy is bounded while squared error is not. 8. Which statements about the softmax classifier and its stable implementation are correct? Select all that apply. - For W of shape (K, D) and Z = X W^T + b, the weight gradient is dW = dZ^T X with shape (K, D). - With one-hot labels, the gradient with respect to the logits is dZ = (P - Y) / N. - A K-class softmax has K independent logit directions. - Subtracting the row maximum before exponentiating prevents overflow without changing the probabilities, because softmax is shift-invariant. - You should compute np.log(softmax(Z)) directly to get the loss. 9. The numerically stable logistic loss, computed from the logit z, is np.logaddexp(0, z) - y * z. Its derivative with respect to z is ____. 10. Derive on paper the gradient of the softmax cross-entropy with respect to the logits, starting from L = -sum_k y_k log p_k and the Jacobian dp_k/dz_j = p_k(delta_kj - p_j). Then write the vectorized train_step for a batch and state every shape. Rate your work against the rubric. --- ## Contents Part of [🧮 Mathematics for Machine Learning 4: Build Core ML Algorithms from Scratch](https://everwise.study/courses/mathematics-for-machine-learning-4-build-core-ml-algorithms-from-scratch.md) - **📈 Linear and Cross-Entropy Models from Scratch** - [📉 Linear Regression with Normal Equations and Gradient Descent](https://everwise.study/courses/mathematics-for-machine-learning-4-build-core-ml-algorithms-from-scratch/linear-regression-normal-equations-and-gradient-descent.md) - [🔍 Gradient Checking with a Finite-Difference Harness](https://everwise.study/courses/mathematics-for-machine-learning-4-build-core-ml-algorithms-from-scratch/gradient-checking-finite-difference-harness.md) (sign-in required) - [🎚️ Logistic Regression to Softmax: Cross-Entropy Done Right](https://everwise.study/courses/mathematics-for-machine-learning-4-build-core-ml-algorithms-from-scratch/logistic-regression-to-softmax-cross-entropy.md) - [✅ Checkpoint: Linear and Cross-Entropy Models](https://everwise.study/courses/mathematics-for-machine-learning-4-build-core-ml-algorithms-from-scratch/checkpoint-linear-and-cross-entropy-models.md) — this lesson - **🔵 Unsupervised Learning from Scratch** - [🧭 PCA via SVD: Explained Variance and Component Selection](https://everwise.study/courses/mathematics-for-machine-learning-4-build-core-ml-algorithms-from-scratch/pca-via-svd-explained-variance.md) - [🌀 k-means: Lloyd's Algorithm and k-means++ Initialization](https://everwise.study/courses/mathematics-for-machine-learning-4-build-core-ml-algorithms-from-scratch/k-means-lloyd-and-k-means-plus-plus.md) - [🔔 Gaussian Mixture Models with Expectation Maximization](https://everwise.study/courses/mathematics-for-machine-learning-4-build-core-ml-algorithms-from-scratch/gaussian-mixture-models-expectation-maximization.md) - [✅ Checkpoint: PCA, k-means and GMM](https://everwise.study/courses/mathematics-for-machine-learning-4-build-core-ml-algorithms-from-scratch/checkpoint-pca-kmeans-and-gmm.md) - **🧠 Backpropagation and Benchmark Capstone** - [🔗 Backpropagation for a One-Hidden-Layer MLP](https://everwise.study/courses/mathematics-for-machine-learning-4-build-core-ml-algorithms-from-scratch/backpropagation-for-one-hidden-layer-mlp.md) - [📊 Cross-Validation and a Fair Baseline Comparison Protocol](https://everwise.study/courses/mathematics-for-machine-learning-4-build-core-ml-algorithms-from-scratch/cross-validation-and-baseline-comparison-protocol.md) - [🏁 Capstone: NumPy Implementations vs Library Baselines](https://everwise.study/courses/mathematics-for-machine-learning-4-build-core-ml-algorithms-from-scratch/capstone-numpy-vs-library-baselines.md) (premium content) - [✅ Checkpoint: Backpropagation and Model Comparison](https://everwise.study/courses/mathematics-for-machine-learning-4-build-core-ml-algorithms-from-scratch/checkpoint-backpropagation-and-model-comparison.md) (premium content) - [Course cheatsheet](https://everwise.study/courses/mathematics-for-machine-learning-4-build-core-ml-algorithms-from-scratch/cheatsheet.md) (premium content)