Checkpoint: Linear and Cross-Entropy Models

By Better Wise

Quiz

You have a full-rank regression problem with n = 500 rows and d = 4 columns (including the intercept). Which solver should you use, and why?
- Batch gradient descent, because the normal equations are always numerically unstable.
- Batch gradient descent, because O(d^3) factorization is prohibitive at d = 4.
- Neither: you must invert X^T X explicitly to get the answer.
- The normal equations via np.linalg.lstsq, because O(nd^2 + d^3) is trivial at d = 4 and it gives the exact optimum in one shot.
Which statements about batch gradient descent for linear regression are correct? Select all that apply.
- Features should be standardized to zero mean and unit variance so a single learning rate suits every coordinate.
- If the loss oscillates or explodes, the learning rate is likely too large.
- One iteration costs O(nd), one pass for the residual and one for the gradient.
- You should monitor the parameter norm, not the loss, to decide convergence.
- On full-rank data, GD should converge to a different optimum than the normal equations.
The gradient of the mean squared error J(w) = (1/2n)||Xw - y||^2 with respect to w is ____.
Your finite-difference check reports a worst relative error of about 0.96, and every coordinate fails by roughly the same factor. What is the most likely bug?
- The check was run at a kink such as ReLU at 0.
- An indexing or slicing error affecting a single coordinate.
- The check was run in float32, so roundoff explains it.
- A scale or normalization bug, such as a missing 1/n on the whole data term of the gradient.
Which practices make a finite-difference gradient check trustworthy? Select all that apply.
- Check at all-zero parameters, since that is the most reproducible point.
- Use the central difference (f(x + eps e_i) - f(x - eps e_i)) / (2 eps), whose truncation error is O(eps^2).
- Compare the sums of the two gradients instead of the entries.
- Compare gradients elementwise with the relative error |num - ana| / max(1e-8, |num| + |ana|).
- Restore the perturbed entry after each coordinate so later evaluations are not corrupted.
For float64 arrays, the standard central-difference step size is about ____ (give the numeric value used in the lesson).
Why does cross-entropy, not squared error, give the clean gradient p - y for a sigmoid output?
- The p(1-p) from the sigmoid derivative cancels exactly against the 1/(p(1-p)) factor in dL/dp.
- Because squared error has no gradient at all for a sigmoid output.
- Because cross-entropy removes the sigmoid from the model.
- Because cross-entropy is bounded while squared error is not.
Which statements about the softmax classifier and its stable implementation are correct? Select all that apply.
- For W of shape (K, D) and Z = X W^T + b, the weight gradient is dW = dZ^T X with shape (K, D).
- With one-hot labels, the gradient with respect to the logits is dZ = (P - Y) / N.
- A K-class softmax has K independent logit directions.
- Subtracting the row maximum before exponentiating prevents overflow without changing the probabilities, because softmax is shift-invariant.
- You should compute np.log(softmax(Z)) directly to get the loss.
The numerically stable logistic loss, computed from the logit z, is np.logaddexp(0, z) - y * z. Its derivative with respect to z is ____.
Derive on paper the gradient of the softmax cross-entropy with respect to the logits, starting from L = -sum_k y_k log p_k and the Jacobian dp_k/dz_j = p_k(delta_kj - p_j). Then write the vectorized train_step for a batch and state every shape. Rate your work against the rubric.