Bias-Variance Tradeoff in Practice

By Better Wise

The previous lesson gave you a recipe for a point estimate: maximum likelihood picks the parameter that makes the observed data most probable, and MAP adds a prior to pull that estimate toward a sensible region. Both answer "what value should the parameter take?" Neither answers the question a stakeholder actually asks: how wrong will this prediction be on apartments we have not seen yet? This lesson answers it by splitting expected squared error into pieces you can reason about separately, then using that split to choose between a straight line and a flexible curve.

• Write expected squared error as irreducible noise + bias² + variance
• Explain why the bias and variance terms move in opposite directions as a model gets more flexible
• Show how sample size changes the price of flexibility
• Choose between a simple and a flexible model using validation error and the one-standard-error rule

The dataset question: rent against apartment size

You have 40 apartments, each with floor area in m² and monthly rent. The question: predict rent for a new apartment from its area. Domain knowledge says the relationship is mildly curved, since rent per m² tends to flatten out for very large units. Two candidates:

• Model A: a straight line (degree 1, two parameters).
• Model B: a degree-9 polynomial (ten parameters).

Fit both on the 40 rows. Model B's training error comes out far lower, always. That is not evidence it predicts better, and the decomposition below explains why.

Where the error on new data comes from

Write the data-generating process as $y = f(x) + \varepsilon$, where $f$ is the truth and $\varepsilon$ is noise with mean 0 and variance $\sigma^2$. You train a model $\hat{f}$ on a random training set $D$. Now imagine repeating the entire experiment many times, each time drawing a new training set, refitting, and scoring at the same fixed area $x_0$. Averaging the squared error over those repetitions gives:

$$ \mathbb{E}\left[(y_0 - \hat{f}(x_0))^2\right] = \sigma^2 + \left(\mathbb{E}[\hat{f}(x_0)] - f(x_0)\right)^2 + \operatorname{Var}\left(\hat{f}(x_0)\right) $$

Read the three pieces in order:

1. Irreducible noise $\sigma^2$. Two apartments of identical size still rent for different amounts. No model removes this.
2. Bias². The gap between the average fitted value across training sets and the truth. A straight line fit to a curved truth has bias at every $x_0$, and collecting more data does not shrink it.
3. Variance. The spread of $\hat{f}(x_0)$ around its own average. It comes from the particular rows that landed in your training set.

Variance scales roughly like $1/n$, while bias² is driven by model shape. That asymmetry is the whole tradeoff.

Worked example: 40 apartments, three models

Simulating many training sets of 40 rows (error in hundreds of dollars squared, $\sigma^2 = 0.20$):

The degree-9 fit has essentially eliminated bias and still loses badly. Ten coefficients pinned by 40 rows means each one is set by a handful of points, so swapping three apartments swings the curve, especially near the edges of the size range. Bias falls quickly as you first add flexibility, variance grows slowly at first, and the sum bottoms out somewhere in the middle. That U-shape is the tradeoff.

Worked example: more data changes the verdict

Same degree-9 model, now trained on 160 apartments. The variance term falls roughly fourfold, from 1.35 to about 0.34, while bias² stays near 0.01. Expected test MSE drops to about 0.55, which now beats the straight line at 0.68. Flexibility is not a fixed property of a model. It is flexibility relative to your sample size. A rough guard: once the effective parameter count approaches a tenth of your rows, variance starts dominating the sum.

Worked example: regularization as a bias dial

Ridge regression shrinks coefficients toward zero, and the shrinkage strength is the same Gaussian prior you met with MAP: MAP with a Gaussian prior is exactly ridge. Turning the penalty up moves you along the tradeoff without changing the model family. On the apartment data, shrinking the degree-9 fit gives bias² 0.10 and variance 0.30, for a total of 0.60, better than either extreme. You are deliberately accepting a little bias to buy a lot of stability.

Choosing between the simple and the flexible model

Training error is monotone in flexibility, so it can never make this choice for you. Compare candidates on held-out data or by k-fold cross-validation, then read the bias-variance split off the two errors you already have.

flowchart TD
    Start["Compare models by validation error"] --> Gap{"Large gap to training error?"}
    Gap -->|"Yes"| Var["📈<br/>High variance<br/>simplify, regularize, or get more rows"]
    Gap -->|"No, both high"| Bias["📉<br/>High bias<br/>more flexibility or better features"]
    Var --> Pick["Pick the simplest model within one SE of the best"]
    Bias --> Pick

If validation error is high and close to training error, extra rows will barely help: you have a bias problem. If the gap is wide, more rows or a penalty is the fix.

Quick recall, fill each blank before moving on:

The term in the decomposition that no amount of data can remove is _____. The squared gap between the average fitted value and the truth is the _____ term. The spread of fitted values across repeated training sets is the _____ term. Training error always _____ as flexibility increases, so it cannot be used to select a model. The one-standard-error rule picks the _____ model within one standard error of the best validation score.

Practical Rules for Choosing Model Flexibility

• Report the split, not just the score. Saying "validation MSE is 0.47, and the training gap is small" tells a colleague whether to collect data or change the model. A bare number does not.
• Common mistake: tuning on the test set. Every time you look at test error and adjust, it leaks into your choice. Tune with cross-validation, touch the test set once.
• Common mistake: trusting the U-curve minimum too literally. Validation error is itself noisy. Within one standard error, take the simpler model; it usually generalizes more predictably.
• Estimate the noise floor. If validation MSE has flattened near $\sigma^2$, stop engineering. You are fighting irreducible noise.
• Small samples punish flexibility fast. With fewer than about ten rows per effective parameter, prefer the simple model or a penalty, and say so explicitly.
• Check whether the model family can represent the pattern at all. No penalty fixes a straight line fit to a genuinely curved truth; that is bias, and only flexibility or better features reduce it.

What to Carry Forward

Expected squared error splits into irreducible noise, bias², and variance. Flexibility buys down bias and pays in variance, and the price falls as $n$ grows, which is why the right model depends on your dataset, not on a ranking of algorithms. Choose by validation error, prefer simplicity within one standard error, and treat regularization as a deliberate bias-for-variance swap.

Next up: validation error is a random quantity, so when one model beats another by 0.03, is that a real difference? Sampling, confidence intervals and hypothesis tests answer exactly that.

Front: Expected squared error decomposition at a fixed x Back: Irreducible noise sigma squared, plus bias squared, plus the variance of the fitted value

Front: Bias of an estimator f-hat at x Back: E[f-hat(x)] minus f(x): the gap between the average fit and the truth

Front: Variance term in the decomposition Back: E[(f-hat(x) - E[f-hat(x)]) squared]: the spread of fits across repeated training sets

Front: Which term shrinks as n grows Back: Variance shrinks roughly like 1/n; bias squared stays roughly fixed

Front: Which term does adding parameters mainly reduce Back: Bias squared, while variance usually rises

Front: Why training error cannot choose a model Back: It decreases monotonically with flexibility and never reveals the variance cost

Front: One-standard-error rule Back: Choose the simplest model whose validation error is within one standard error of the best

Front: A wide gap between training and validation error signals what Back: High variance: simplify, regularize, or collect more rows

Front: High training error and high validation error together signal what Back: High bias: the model is too rigid, so add flexibility or better features

Front: What ridge regression does to the bias-variance split Back: It raises bias and lowers variance by shrinking coefficients; it is MAP with a Gaussian prior

Front: What sets the floor on achievable expected error Back: The irreducible noise sigma squared

Front: Typical shape of validation error as flexibility increases Back: U-shaped: bias falls, variance rises, and the sum bottoms out in the middle

Front: Effect of training a 10-parameter polynomial on 160 rows instead of 40 Back: Variance falls roughly like 1/n, here from about 1.35 to about 0.34, so total error drops