Covariance, Correlation and Dependence

By Better Wise

Last lesson you measured how one column spreads out with expectation and variance. Most real dataset questions are about two columns at once: does temperature move daily ride counts, does discount depth move basket size? Covariance is just variance's two-column cousin, and correlation is that number with the units cancelled so you can compare it across questions. By the end of this lesson you should be able to:

• compute the covariance of two small columns by hand;
• turn it into a Pearson correlation and say what the standardisation bought you;
• explain, with a counterexample, why a correlation near zero does not prove independence;
• recognise the three situations where a correlation number misleads.

Covariance asks whether two columns deviate together

For a single variable, variance averages the squared deviation from the mean. For two variables, covariance averages the product of deviations:

$$ \text{Cov}(X,Y) = E\big[(X - E[X])(Y - E[Y])\big] = E[XY] - E[X],E[Y] $$

The sign is the message. If days above average temperature also tend to be days above average rides, the products are mostly positive and the covariance is positive. If one column rises while the other falls, the products are mostly negative. If the columns drift independently, the signs cancel.

Worked example. Five days of a bike-share dataset, with temp $X$ in °C and rides $Y$ in hundreds:

$E[X] = 18$, $E[Y] = 6$, and the products sum to 64, so $\text{Cov}(X,Y) = 64/5 = 12.8$.

Now look at the unit: it is degrees-Celsius times hundreds-of-rides. That is not a quantity anyone can interpret, and it changes if you switch to Fahrenheit or count rides one by one. Covariance carries the scale of both columns.

Correlation divides out the scale

Divide the covariance by the two standard deviations and the units cancel:

$$ r = \frac{\text{Cov}(X,Y)}{\sigma_X \sigma_Y} $$

Continuing the example: $\sigma_X = \sqrt{32} = 5.66$ and $\sigma_Y = \sqrt{5.2} = 2.28$, so

$$ r = \frac{64}{\sqrt{160 \times 26}} = \frac{64}{64.5} \approx 0.992 $$

Properties worth memorising: $r$ always lies in $[-1, 1]$; it is unchanged if you rescale either column (Celsius to Fahrenheit, rides to thousands); swapping the columns does not change it; and using the sample divisor $n-1$ instead of $n$ changes neither $r$ nor its interpretation. Two related identities you will reuse constantly: $\text{Cov}(X,X) = \text{Var}(X)$, and

$$ \text{Var}(X + Y) = \text{Var}(X) + \text{Var}(Y) + 2,\text{Cov}(X,Y) $$

Zero correlation is not independence

Two facts sit side by side, and only one of them is reversible:

• If $X$ and $Y$ are independent, then $\text{Cov}(X,Y) = 0$ (because $E[XY] = E[X]E[Y]$).
• If $\text{Cov}(X,Y) = 0$, they need not be independent.

Counterexample. Let $X$ take the values $-1, 0, 1$ with equal probability, and let $Y = X^2$. Then $E[X] = 0$ and $E[XY] = E[X^3] = 0$, so $\text{Cov}(X,Y) = 0$ and $r = 0$. Yet $Y$ is completely determined by $X$: knowing $X$ tells you $Y$ exactly. Zero correlation rules out linear association, nothing more.

Three ways a correlation number misleads

Nonlinear but strong. The counterexample above is a perfect U-shape in disguise. Any symmetric curve gives $r \approx 0$ while the relationship is deterministic. Fix: plot the scatter, and compare means of $Y$ across bins of $X$.

One high-leverage point. In an ad-spend versus revenue table of 40 stores with $r = 0.1$, adding a single flagship store with extreme spend and extreme revenue can push $r$ to $0.7$. Correlation is not robust, so always look before you trust.

Restricted range. Across all 500 listings, size and price correlate at $r = 0.65$. Inside one neighbourhood where sizes cluster between 60 and 75 m², the same data gives $r = 0.15$. Neither number is wrong: you measured different populations. This is the same effect that will reappear as a bias concern later in the course.

Exercises on a second dataset

An e-commerce table has discount_pct and basket_value (in dollars).

1. You compute $\text{Cov}(\text{discount}, \text{basket}) = -4.5$, with $\sigma_{\text{discount}} = 3$ and $\sigma_{\text{basket}} = 2.5$. What is $r$, and how do you read it?
2. A colleague reports $\text{Var}(X) = 4$, $\text{Var}(Y) = 9$, $\text{Cov}(X,Y) = 3$. What is $\text{Var}(X - Y)$?
3. A model team reports $\text{Cov}(\text{age}, \text{spend}) = 0$ and concludes the two columns are independent. Give one concrete way this conclusion could be wrong.

Show the answers

1. $r = -4.5 / (3 \times 2.5) = -0.6$. A moderate negative linear association: deeper discounts go with smaller baskets, or the other way round depending on column order, which does not affect $r$.
2. $\text{Var}(X - Y) = \text{Var}(X) + \text{Var}(Y) - 2,\text{Cov}(X,Y) = 4 + 9 - 6 = 7$.
3. If spend peaks in middle age and is lower for both young and old customers, the positive and negative deviations cancel and covariance is zero while spend still depends on age. Binning age and comparing mean spend would expose it.

Habits that keep a correlation honest

• Never report $r$ without the $n$ and a scatterplot. Both cost one line and catch most embarrassments.
• A covariance matrix is only comparable within the same pair of units. Rescale a column and every entry touching it changes.
• When $r$ is near zero, ask "could this be a curve?" before writing "no relationship". Bin one column and compare group means.
• Check for a single dominant point and for mixed subgroups before you interpret the sign.
• Correlation is not causation: temperature and ice-cream sales correlate because both follow the season, not because one causes the other.

What to carry into the checkpoint

• Covariance captures joint deviation but carries the product of both units; correlation divides by both standard deviations and lands in $[-1, 1]$.
• Correlation measures linear association only, so a value near zero is weak evidence against dependence and no evidence at all against a curved relationship.
• The identities $\text{Cov}(X,X) = \text{Var}(X)$ and $\text{Var}(X+Y) = \text{Var}(X) + \text{Var}(Y) + 2\text{Cov}(X,Y)$ connect this lesson to the variance work you just finished.

In the next lesson, the Module 1 checkpoint, you put events, conditional probability, expectation, variance and covariance together on one dataset walkthrough.

Covariance is defined as _____, which is the average product of the deviations from the means. Pearson correlation is the covariance divided by _____. A correlation of zero means there is no _____ association between the two columns. Independence implies _____, but zero covariance does not imply independence. The covariance of a variable with itself, Cov(X, X), equals _____.

Front: Formula for Cov(X,Y) Back: E[(X - E[X])(Y - E[Y])] = E[XY] - E[X]E[Y]

Front: What does a positive covariance tell you? Back: Both columns tend to sit above their means together and below their means together.

Front: Why is a raw covariance value hard to interpret? Back: Its unit is the product of the two variables' units, so it changes when either column is rescaled.

Front: Formula for Pearson correlation Back: r = Cov(X,Y) / (sigma_X * sigma_Y)

Front: What range does Pearson r take? Back: Between -1 and 1 inclusive.

Front: Does r change if you convert Celsius to Fahrenheit? Back: No, correlation is unit-free; the covariance does change.

Front: Does r = 0 imply independence? Back: No, it only means there is no linear association.

Front: Which direction is guaranteed: independence giving zero covariance, or zero covariance giving independence? Back: Independence guarantees zero covariance; the reverse fails.

Front: Give a counterexample where Cov(X,Y) = 0 but Y depends on X Back: X uniform on {-1, 0, 1} and Y = X squared: E[X^3] = 0, yet Y is determined by X.

Front: What is Cov(X,X)? Back: Var(X).

Front: Variance of a sum in terms of covariance Back: Var(X + Y) = Var(X) + Var(Y) + 2 Cov(X,Y)

Front: What does restricting the range of a column do to r? Back: It shrinks r toward zero, even if the full-population association is strong.

Front: Effect of one high-leverage outlier on r Back: It can move r a lot, which is why you always plot the scatter first.