📊 Expectation and Variance on a Real Dataset

By Better Wise

In the previous lesson you built a sample space out of sessions.csv and computed conditional probabilities such as P(order | email). Now you compress that same table into two numbers: one for the level of order_value and one for how far a typical row strays from that level. Every model in this course starts from that pair.

By the end of this lesson you can:

  • turn a column into a random variable and read off its empirical distribution;
  • compute the mean, variance and standard deviation by hand on an 8-row table;
  • explain each of the three numbers as an expectation taken over that random variable;
  • apply the shift and scale rules instead of recomputing from rows.

Eight rows, one random variable

Here is the working extract. Eight sessions, three channels, one numeric column.

user_id channel order_value
1 search 20
2 search 40
3 email 10
4 social 0
5 search 30
6 email 0
7 social 50
8 search 10

The question a stakeholder actually asks is "what does a session bring in, typically, and how much does that bounce around?" To answer it probabilistically, define a random variable X: draw one row of the file uniformly at random, set X to that row's order_value. Because each row has probability 1/8, the probability that X takes a given value is just the share of rows holding it.

x rows P(X = x) x · P(X = x) x² · P(X = x)
0 2 0.250 0.00 0.00
10 2 0.250 2.50 25.00
20 1 0.125 2.50 50.00
30 1 0.125 3.75 112.50
40 1 0.125 5.00 200.00
50 1 0.125 6.25 312.50
total 8 1.000 20.00 700.00

That table is the whole lesson. The last two columns are the two expectations you need.

The mean is an expectation

For a discrete random variable, $E[X] = \sum_x x \cdot P(X = x)$. Read the fourth column of the table: the sum is 20, so $E[X] = 20$ dollars.

Sign in to view the full content of this lesson. Sign in

The shortcut sum(order_value) / n = 160 / 8 = 20 gives the same answer, and it is not a coincidence. Averaging rows is exactly the same as weighting each distinct value by its share of rows. The expectation form is the one that survives when the data is not uniform, for example when you weight by traffic share instead of by row count.

Note: The mean is not "the value you expect to see on the next draw". Twenty dollars happens to appear once in this file, but in the email group (values 0 and 10) the mean of 5 is a number no session ever produced. The mean is the long-run average of repeated draws, nothing more.

Spread measured as an expected squared distance

The mean alone hides a real difference. Two files can both average 20 while one is all 20s and the other swings from 0 to 50. Variance answers "how far does X typically land from E[X]?"

$Var(X) = E[(X - E[X])^2]$. Step by step on the raw rows: deviations from 20 are 0, 20, -10, -20, 10, -20, 30, -10, and their squares are 0, 400, 100, 400, 100, 400, 900, 100, which sum to 2400. Divide by 8 and you get 300 dollars squared.

The shortcut uses the fifth column: $Var(X) = E[X^2] - (E[X])^2 = 700 - 20^2 = 700 - 400 = 300$. Same number, one pass over the table, and it is the form that stays cheap when you later stack many columns together.

Variance is the expectation of the squared distance from the mean, which is why it is in dollars squared and not in dollars.

Two denominators appear in practice. Dividing by n gives the population variance (300 here). Dividing by n minus 1 gives the sample variance, 2400 / 7 ≈ 342.9, which is slightly larger because the sample mean is itself estimated from the same rows. Use n minus 1 when the column is a sample you want to generalize from. You will see the precise reason in Module 3.

Standard deviation puts spread back in dollars

Take the square root: $\sqrt{300} \approx 17.32$ dollars. That is the standard deviation, and it is the number you quote to a human, because it lives on the same scale as the column. A sentence like "average order value is 20 dollars, give or take about 17" is immediately meaningful; "variance is 300" is not.

Also note what 17.32 tells you: with a mean of 20 and a spread of 17, orders of 0 and 50 are ordinary, not outliers. A column whose mean is 20 and whose standard deviation is 2 is a very different business.

Conditional means and the total expectation check

The channel column is still there, and the previous lesson's conditioning applies unchanged. Compute the mean inside each group:

  • search: values 20, 40, 30, 10, so E[X | channel = search] = 25;
  • email: values 10, 0, so E[X | channel = email] = 5;
  • social: values 0, 50, so E[X | channel = social] = 25.

Weight each group mean by its share of rows and the overall mean returns: 0.5 · 25 + 0.25 · 5 + 0.25 · 25 = 12.5 + 1.25 + 6.25 = 20. This is the law of total expectation, and it is the cheap sanity check that your group-by query and your overall query agree.

Shifting and scaling a column

Suppose finance reports net value as 0.9 * order_value - 2, where 2 dollars covers fulfillment and 10 percent is the platform fee. You do not need to rebuild the table. For constants a and b:

  • $E[aX + b] = a \cdot E[X] + b$, so here $0.9 \cdot 20 - 2 = 16$ dollars;
  • $Var(aX + b) = a^2 \cdot Var(X)$, so here $0.81 \cdot 300 = 243$, and the standard deviation is $\sqrt{243} \approx 15.59$ dollars.

Adding a constant moves every value but moves none of them apart, so variance ignores b. Scaling by a stretches every gap by a, and squared gaps by a squared.

Check yourself on three short tasks

Try each before opening the answers.

  1. Compute E[X] and Var(X) for the email group only (values 10 and 0).
  2. sessions_count has mean 3 and standard deviation 1.5. What are the mean and standard deviation of 4 * sessions_count + 5?
  3. You must report "typical order value" for a dashboard. Why quote the mean together with the standard deviation rather than the mean alone?

Show the answers

  1. E[X] = (10 + 0) / 2 = 5. E[X²] = (100 + 0) / 2 = 50, so Var(X) = 50 − 25 = 25 and the standard deviation is 5 dollars. Note that 5 is not an observed value.
  2. Mean = 4 · 3 + 5 = 17. Variance = 16 · 2.25 = 36, so the standard deviation is 4 · 1.5 = 6.
  3. The mean is a level; the standard deviation is the size of the typical miss. Quoting "20 dollars, standard deviation 17" tells the reader that 0-dollar and 50-dollar sessions are normal, while "20 dollars, standard deviation 2" would signal that something is off.

Rules of thumb that prevent silent errors

  • Always attach units. Dollars for the mean and the standard deviation, dollars squared for the variance. If a variance comes out smaller than the mean for a strictly positive column, check whether you actually computed a standard deviation.
  • Prefer E[X²] − (E[X])² in code: one pass over the column, and it extends directly to covariance in the next lesson.
  • Do not compare variances across columns with different units (dollars versus minutes). Compare standard deviations, or ratios like the coefficient of variation, sd divided by mean.
  • A mean of 20 with two rows at 0 is not the same business as a mean of 20 with no zeros. Look at the distribution before you trust the summary.
  • Watch the denominator. ddof=0 in NumPy or STDDEV_POP in SQL divides by n; ddof=1 and STDDEV_SAMP divide by n minus 1. Mixing them silently gives slightly different numbers in reports.
  • Sanity check any group-by mean against the law of total expectation. If the weighted group means do not return the overall mean, your weights or your filter are wrong.

What to carry into the next lesson

The two numbers you computed, E[X] = 20 and sd(X) ≈ 17.32, are the entire description of a single column. But order_value and sessions_count do not move independently: heavy users tend to place bigger orders, and that joint behavior needs a third summary number. The next lesson, Covariance, Correlation and Dependence, builds exactly that, using the same E[X²] − (E[X])² machinery you just applied, extended to two columns at once.

Front: How do you turn a data column into a random variable? Back: Draw one row uniformly at random and take that row's column value.

Front: What is the expectation of a discrete random variable? Back: E[X] = the sum over x of x times P(X = x), a probability-weighted average.

Front: Why does averaging rows give the same answer as the expectation formula? Back: Each distinct value is effectively weighted by its share of rows, which is its empirical probability.

Front: Mean of order_value in the 8-row sessions.csv extract? Back: 20 dollars.

Front: Definition of variance in expectation form? Back: Var(X) = E[(X minus E[X]) squared], the expected squared distance from the mean.

Front: Computational shortcut for variance? Back: Var(X) = E[X squared] minus (E[X]) squared.

Front: Value of E[X squared] for order_value in the extract? Back: 700 dollars squared.

Front: Population variance of order_value in the extract (divide by n)? Back: 300 dollars squared.

Front: Standard deviation of order_value in the extract? Back: About 17.32 dollars, the square root of 300.

Front: Why quote standard deviation instead of variance? Back: It is on the original scale, dollars rather than dollars squared, so people can read it.

Front: What does dividing by n minus 1 instead of n do? Back: It gives the sample variance, slightly larger, correcting a downward bias.

Front: State the law of total expectation. Back: E[X] equals the sum over groups g of P(g) times E[X given g].

Front: E[order_value given channel = search] in the extract? Back: 25 dollars, from values 20, 40, 30 and 10 over four rows.

Front: How do constants a and b change the mean and variance? Back: E[aX + b] = a E[X] + b and Var(aX + b) = a squared times Var(X).

  1. For a discrete random variable, the expectation is _____.
  2. The mean order value in the sessions.csv extract is _____ dollars.
  3. The population variance of order_value in that extract is _____ dollars squared.
  4. The standard deviation is the square root of the variance, so it is measured in _____.
  5. For any constants a and b, the variance of aX plus b equals _____.
  6. Dividing the squared deviations by n minus 1 gives the _____.