# 📊 Cross-Validation and a Fair Baseline Comparison Protocol

By Better Wise

Your MLP from the last lesson passes the finite-difference gradient check, which means it computes gradients correctly. Correctness of gradients is not the same as a fair claim of parity with a library. This lesson sets up the protocol that decides whether "my NumPy model matches scikit-learn" is a statement you can defend or just a lucky seed.

## Protocol skills you will leave with

- Build a stratified k-fold splitter that hands *identical* folds to your implementation and to the library baseline.
- Keep preprocessing, tuning, and early stopping inside the fold so nothing leaks.
- Compare models on paired per-fold differences, with a confidence interval, instead of eyeballing two means.
- Spot the three ways benchmark comparisons usually cheat: leakage, unequal tuning budget, wrong noise model.

1. Fix the split object once (same indices for every model).
2. Fit every transform on the training fold only.
3. Give each model the same tuning budget or none at all.
4. Record one score per fold per model.
5. Compare paired differences, not marginal means.
6. Repeat the whole CV with several split seeds and report spread.

## Worked example: one protocol, two logistic regressions

Say your `MyLogReg` (from Module 1) is the challenger and `sklearn.linear_model.LogisticRegression` is the baseline. The harness below is the unit of work: it takes a factory, so each fold gets a freshly constructed model.

```python
import numpy as np
from sklearn.model_selection import StratifiedKFold
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import log_loss

def cv_scores(factory, X, y, cv):
    scores = []
    for tr, te in cv.split(X, y):            # same indices for every model
        sc = StandardScaler().fit(X[tr])     # fit on train fold only
        Xtr, Xte = sc.transform(X[tr]), sc.transform(X[te])
        m = factory().fit(Xtr, y[tr])
        scores.append(log_loss(y[te], m.predict_proba(Xte)[:, 1]))
    return np.array(scores)

cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=0)
a = cv_scores(lambda: MyLogReg(lr=0.1, epochs=400), X, y, cv)
b = cv_scores(lambda: LogisticRegression(max_iter=5000), X, y, cv)

d = a - b
se = d.std(ddof=1) / np.sqrt(d.size)
print(f"mean diff {d.mean():.4f} +/- {1.96 * se:.4f}")
```

Four things make this fair, and each one is a decision you have to defend in the capstone write-up.

1. **Same folds.** `cv` is built once and passed to both calls. If you instead create `StratifiedKFold` separately inside each call and forget `random_state`, the two models see different data and every comparison becomes unpaired noise.
2. **Same preprocessing.** The scaler is fit on `X[tr]`. Fit it on all of `X` and the test fold's mean and variance leak into training, inflating both models and usually the challenger.
3. **Same convergence.** `max_iter=5000` for sklearn exists because its default would stop early and lose to your 400-epoch run for the wrong reason. Check `n_iter_` and your own loss curve. A model that did not converge is not a baseline.
4. **Same metric code.** Compute log loss from `predict_proba` for both. If your NumPy model returns hard labels and sklearn returns probabilities, you are comparing accuracy to log loss.

The `d` vector is the point of the whole exercise. Both models saw fold `k` under identical conditions, so `a[k] - b[k]` cancels the fold's difficulty and isolates the model difference.

## Practice: audit and repair a comparison protocol

Use your own repository for these. Write the answer before you run anything.

**Drill 1 (leakage).** A teammate's script does `StandardScaler().fit_transform(X)` on the full dataset, then calls `cross_val_score`. Their NumPy model scores 0.03 better than sklearn. Is that gap real? Rewrite the two lines that fix it.

**Drill 2 (fold design).** You have 120 samples and 3 classes with counts 90/24/6. You want 5-fold CV for a softmax classifier. Which splitter do you use, and how many examples of the smallest class land in each test fold? What breaks with `n_splits=10`?

**Drill 3 (paired read).** Five folds give mean log loss 0.412 for NumPy and 0.398 for sklearn, with per-fold differences having sample standard deviation 0.020. Compute the standard error and a 95% interval (t critical at df=4 is 2.776). Is the challenger worse in any way you would report as a result?

### Worked answers for the three audit drills

**Drill 1.** The gap is not real. Fitting the scaler on the full dataset leaks test statistics into training, and `cross_val_score` will not catch it because the leak happened before the split. Fix: pass a `Pipeline([("sc", StandardScaler()), ("clf", model)])` to `cross_val_score`, or copy the `cv_scores` harness above, where `sc.fit` only ever sees `X[tr]`.

**Drill 2.** Use `StratifiedKFold(n_splits=5, shuffle=True, random_state=...)`. With stratification the smallest class contributes 6/5, so one or two test folds get 1 example and the rest get 1 or 2 (6 = 2+1+1+1+1). Its per-fold log loss will be jumpy, so also report macro-F1 or per-class recall. With `n_splits=10` some folds can get zero examples of the smallest class, and a fold with no class-2 sample makes macro-averaged metrics undefined. 5 folds is the practical maximum here.

**Drill 3.** se = 0.020 / √5 ≈ 0.0089. Interval: 0.014 ± 2.776 × 0.0089 ≈ 0.014 ± 0.025, so roughly [-0.011, 0.039]. The interval contains zero, so with five folds you cannot claim the NumPy model is worse. What you can report: "no detectable difference on 5 folds; the study is underpowered to resolve gaps below about 0.025 log loss". Then add `RepeatedStratifiedKFold(n_splits=5, n_repeats=5)` to tighten the interval, and remember repeats are correlated, so treat the resulting interval as indicative, not exact.

## Where fair comparisons quietly break

- **Unequal tuning budget.** If you grid-searched your NumPy model's learning rate but ran sklearn at defaults, you compared tuning effort, not implementations. Either tune both with the same grid size and the same inner folds, or run both at fixed, documented settings.
- **Tuning on the reported folds.** Selecting the best hyperparameter on the outer folds and then reporting that score is optimistic for both models and hides which one overfits the selection. Use nested CV: inner folds pick hyperparameters, outer folds report.
- **Timing claims without conditions.** sklearn dispatches to multithreaded BLAS; a Python loop in your k-means is single-threaded. Record hardware, thread count, and warm-up policy, or drop the timing column. If you keep it, time 5 runs per fold and report the median.
- **Unsupervised metrics.** For PCA, k-means, and GMM there are no labels, so held-out evaluation means: fit the transformer on the train fold, `transform` the test fold, and score held-out reconstruction error or held-out mean log-likelihood. Never fit PCA on the full matrix and then split.
- **One seed.** A single split gives you one number with no uncertainty. Three to five repeats is the cheapest credibility you can buy.

## Protocol rules I enforce on every benchmark run

- Freeze the split indices once and save them. Both models read the same array.
- Every fitted object that touches data goes inside the fold loop.
- Report the per-fold table, then the mean, then the paired difference with its interval. Never the mean alone.
- If the interval crosses zero, write "no detectable difference". That is a valid, publishable result.
- Never compare a converged model to an unconverged one, even if the numbers look good.
- Log the versions: `numpy`, `scikit-learn`, Python, BLAS backend, and the exact git commit of your implementation.

## What to carry into the capstone

You now have a protocol, not just a script: identical stratified folds, in-fold preprocessing, matched budgets, and paired intervals. _____ keeps class proportions stable per fold, which matters most when the _____ is tiny. Any transform must be fit on the _____ only. Because both models see the _____, you compare _____, not two independent means. Selecting hyperparameters on the folds you report gives an _____ score, so nested CV separates selection from evaluation.

Front: Why must the split object be created once and reused for both models?
Back: Identical fold indices make the per-fold scores paired, so fold difficulty cancels in the difference.

Front: What does StratifiedKFold guarantee?
Back: Each fold preserves approximately the same class proportion as the full dataset.

Front: Where must StandardScaler be fit in a CV loop?
Back: On the training fold only, then applied to the test fold.

Front: What is the correct statistic for comparing two models on the same folds?
Back: The mean of the paired per-fold differences, with a standard error or confidence interval.

Front: When is a difference between two models not reportable as a result?
Back: When the confidence interval on the paired difference contains zero.

Front: Why can unequal max_iter or epoch counts invalidate a comparison?
Back: An unconverged model loses for a reason unrelated to the implementation being compared.

Front: What does nested CV separate?
Back: Hyperparameter selection (inner folds) from performance estimation (outer folds).

Front: How do you cross-validate PCA or k-means?
Back: Fit on the train fold, transform or assign on the test fold, score held-out reconstruction error or log-likelihood.

Front: What is the risk of a single 5-fold split?
Back: One number with no uncertainty; small differences can be pure split noise. Use repeated CV.

Front: Why is wall-clock timing an unfair column by default?
Back: sklearn uses multithreaded BLAS while a NumPy loop is serial, so hardware and thread count must be reported or timing dropped.

Front: With class counts 90/24/6 and 5 folds, what problem appears in metrics?
Back: Folds with 1 example of the smallest class make macro-averaged metrics unstable.

Front: What must be logged alongside the scores for a reproducible benchmark?
Back: Library versions, BLAS backend, hardware, split seeds, and the git commit.

Next lesson takes this harness and runs the full capstone: your linear model, softmax classifier, PCA, k-means, GMM, and MLP, each scored against its scikit-learn counterpart in a single results table you can defend line by line.

---

## Contents

Part of [🧮 Mathematics for Machine Learning 4: Build Core ML Algorithms from Scratch](https://everwise.study/courses/mathematics-for-machine-learning-4-build-core-ml-algorithms-from-scratch.md)

- **📈 Linear and Cross-Entropy Models from Scratch**
  - [📉 Linear Regression with Normal Equations and Gradient Descent](https://everwise.study/courses/mathematics-for-machine-learning-4-build-core-ml-algorithms-from-scratch/linear-regression-normal-equations-and-gradient-descent.md)
  - [🔍 Gradient Checking with a Finite-Difference Harness](https://everwise.study/courses/mathematics-for-machine-learning-4-build-core-ml-algorithms-from-scratch/gradient-checking-finite-difference-harness.md) (sign-in required)
  - [🎚️ Logistic Regression to Softmax: Cross-Entropy Done Right](https://everwise.study/courses/mathematics-for-machine-learning-4-build-core-ml-algorithms-from-scratch/logistic-regression-to-softmax-cross-entropy.md)
  - [✅ Checkpoint: Linear and Cross-Entropy Models](https://everwise.study/courses/mathematics-for-machine-learning-4-build-core-ml-algorithms-from-scratch/checkpoint-linear-and-cross-entropy-models.md)
- **🔵 Unsupervised Learning from Scratch**
  - [🧭 PCA via SVD: Explained Variance and Component Selection](https://everwise.study/courses/mathematics-for-machine-learning-4-build-core-ml-algorithms-from-scratch/pca-via-svd-explained-variance.md)
  - [🌀 k-means: Lloyd's Algorithm and k-means++ Initialization](https://everwise.study/courses/mathematics-for-machine-learning-4-build-core-ml-algorithms-from-scratch/k-means-lloyd-and-k-means-plus-plus.md)
  - [🔔 Gaussian Mixture Models with Expectation Maximization](https://everwise.study/courses/mathematics-for-machine-learning-4-build-core-ml-algorithms-from-scratch/gaussian-mixture-models-expectation-maximization.md)
  - [✅ Checkpoint: PCA, k-means and GMM](https://everwise.study/courses/mathematics-for-machine-learning-4-build-core-ml-algorithms-from-scratch/checkpoint-pca-kmeans-and-gmm.md)
- **🧠 Backpropagation and Benchmark Capstone**
  - [🔗 Backpropagation for a One-Hidden-Layer MLP](https://everwise.study/courses/mathematics-for-machine-learning-4-build-core-ml-algorithms-from-scratch/backpropagation-for-one-hidden-layer-mlp.md)
  - [📊 Cross-Validation and a Fair Baseline Comparison Protocol](https://everwise.study/courses/mathematics-for-machine-learning-4-build-core-ml-algorithms-from-scratch/cross-validation-and-baseline-comparison-protocol.md) — this lesson
  - [🏁 Capstone: NumPy Implementations vs Library Baselines](https://everwise.study/courses/mathematics-for-machine-learning-4-build-core-ml-algorithms-from-scratch/capstone-numpy-vs-library-baselines.md) (premium content)
  - [✅ Checkpoint: Backpropagation and Model Comparison](https://everwise.study/courses/mathematics-for-machine-learning-4-build-core-ml-algorithms-from-scratch/checkpoint-backpropagation-and-model-comparison.md) (premium content)
- [Course cheatsheet](https://everwise.study/courses/mathematics-for-machine-learning-4-build-core-ml-algorithms-from-scratch/cheatsheet.md) (premium content)