# 🧠 Bayes' Rule and Base Rates

By Better Wise

You now have the pieces: conditional probability from Module 1, and the Gaussian and exponential models from the last lesson. Bayes' rule is where those pieces pay off. It is the tool that turns "how likely is this evidence if the claim is true?" into "how likely is the claim, given the evidence?" Those two questions are not the same, and confusing them is the most common reasoning error in data work.

By the end of this lesson you can:

- Write Bayes' rule and name each of its four parts
- Apply it to a base-rate question with real numbers
- Explain why the posterior is far smaller than most first guesses
- Use the odds form to update a belief quickly
- Spot base-rate neglect in a reported metric

## The four parts of the update

For a hypothesis $H$ and observed evidence $E$:

$$
P(H \mid E) = \frac{P(E \mid H)\,P(H)}{P(E)}
$$

| Part | Name | Plain meaning |
| --- | --- | --- |
| $P(H)$ | prior | what you believed before seeing the evidence |
| $P(E \mid H)$ | likelihood | how well the hypothesis predicts the evidence |
| $P(E)$ | marginal likelihood | how often the evidence occurs overall |
| $P(H \mid E)$ | posterior | what you should believe now |

The denominator expands to $P(E \mid H)P(H) + P(E \mid \lnot H)P(\lnot H)$, which is just the law of total probability. That expansion is the whole engine of base-rate reasoning: the evidence has to be explained under *both* hypotheses before you can credit either one.

## The base-rate question: a fraud screen

A payments team ships a rule that flags suspicious transactions. On historical data:

- 1% of transactions are actually fraudulent. So $P(F) = 0.01$.
- The rule catches 90% of fraud. So $P(\text{flag} \mid F) = 0.90$.
- The rule falsely flags 5% of clean transactions. So $P(\text{flag} \mid \lnot F) = 0.05$.

A transaction is flagged. What is the probability it is fraudulent?

Most people answer "about 90%". Let's compute. Imagine 10,000 transactions:

| | Flagged | Not flagged | Total |
| --- | --- | --- | --- |
| Fraud | 90 | 10 | 100 |
| Clean | 495 | 9,405 | 9,900 |
| Total | 585 | 9,415 | 10,000 |

Of the 585 flags, only 90 are genuine. So

$$
P(F \mid \text{flag}) = \frac{90}{585} \approx 0.154
$$

The posterior is about **15%**, not 90%. The false positives (495) outnumber the true positives (90) by more than five to one, simply because clean transactions are 99 times more common. **A rare hypothesis needs much stronger evidence to reach a high posterior.**

Note the pattern: the numerator is one cell of the table, the denominator is its row-and-column total. Every Bayes problem on a 2x2 table is a cell over a marginal.

## Why intuition fails here

The confusion is a swap of conditioning direction:

- $P(\text{flag} \mid F) = 0.90$ is a property of the *test*.
- $P(F \mid \text{flag}) = 0.15$ is a property of the *world after the test*.

These differ whenever the prior is not 50/50. Sensitivity tells you how the test behaves on known fraud; it says nothing about the mix of traffic the test sees. Base-rate neglect is the habit of answering the second question with the first number.

Vary the prior and watch the posterior move while the test stays identical:

| Prior $P(F)$ | Posterior $P(F \mid \text{flag})$ |
| --- | --- |
| 0.01 | 0.154 |
| 0.10 | 0.667 |
| 0.50 | 0.947 |

Same 90% sensitivity, same 5% false positive rate. Only the base rate changed, and the answer went from "probably not" to "almost certainly".

## The odds form: updating without denominators

Divide the posterior for $H$ by the posterior for $\lnot H$ and $P(E)$ cancels:

$$
\frac{P(H \mid E)}{P(\lnot H \mid E)} = \frac{P(H)}{P(\lnot H)} \times \frac{P(E \mid H)}{P(E \mid \lnot H)}
$$

**posterior odds = prior odds × likelihood ratio**

For the fraud screen: prior odds $0.01/0.99 \approx 0.0101$, likelihood ratio $0.90/0.05 = 18$. Posterior odds $\approx 0.182$, which converts to probability $0.182/(1+0.182) \approx 0.154$. Same answer, one multiplication, no table.

This form is what makes sequential updates easy. Two independent tests, each with likelihood ratio 18, multiply to 324, pushing odds from 0.0101 to about 3.27, or a posterior near 77%. Evidence accumulates multiplicatively; beliefs move fast only when evidence is strong relative to how rare the hypothesis was.

## Worked example: a sensor on a pipeline

A pressure sensor reports a leak with probability 0.98 when a leak exists, and 0.02 when it does not. Historically, 0.5% of hourly readings are taken during a real leak.

Prior odds: $0.005/0.995 \approx 0.00503$. Likelihood ratio: $0.98/0.02 = 49$. Posterior odds $\approx 0.246$, so $P(\text{leak} \mid \text{alarm}) \approx 0.198$, roughly 20%.

Now the operational point: if an alarm triggers a shutdown costing $5,000, and a missed leak costs $200,000, compare expected costs. Shutting down every alarm costs $5,000 per alarm; ignoring alarms costs $200,000 \times 0.198 \approx $39,600$ per alarm in expected leakage. The alarm is still worth acting on at 20% posterior, because the costs are asymmetric. A low posterior does not mean the evidence is useless; it means you weigh it against what each decision costs.

## Where the posterior number comes from

The two inputs that carry the most weight are often the ones nobody measures carefully:

- **The prior.** Here it came from historical fraud rates. If fraud patterns shifted, the prior is stale and the posterior is wrong, no matter how good the test is.
- **The false positive rate.** False positives multiply against a huge denominator of negatives. Halving the false positive rate from 5% to 2.5% moves the fraud posterior from 15.4% to about 26.7%, a bigger gain than improving sensitivity from 90% to 95% would give.

## Practical rules for base-rate reasoning

- **Always ask for the base rate first.** Before any test number, find out how common the thing is. If a precision or recall figure is quoted without a prevalence, the conversion to a real-world probability is incomplete.
- **Precision is a posterior; recall is a likelihood.** Precision answers $P(\text{true} \mid \text{flagged})$; recall answers $P(\text{flagged} \mid \text{true})$. Do not read a recall number as if it were precision.
- **Do the table, not the story.** Build the 10,000-row table or the odds multiplication. Fluent intuition is exactly what base-rate neglect exploits.
- **Small shifts in the prior matter enormously at low prevalence.** A prior of 1% versus 2% changes the fraud posterior from 15% to 27%. State your prior and where it came from.
- **Do not round the prior away.** Writing $P(F) = 0$ for a rare event makes the posterior exactly 0 and no evidence can recover it. Rare is not impossible.

Quick self-check: a classifier with 99% precision and 1% prevalence flags a record. Before computing anything, predict whether the posterior is above or below 50%, then verify with the odds form.

## What to carry forward

Bayes' rule combines a prior with a likelihood to produce a posterior, and at low base rates the posterior can be an order of magnitude smaller than the sensitivity figure suggests. The odds form (prior odds × likelihood ratio) is the fastest way to update, and it is the same machinery you will reuse whenever a belief needs revising on new data.

In the next lesson, the checkpoint for this module, you will choose between Bernoulli, binomial, Poisson, Gaussian and exponential for a given dataset question, and run a full Bayes update end to end.

Front: State Bayes' rule for a hypothesis H and evidence E.
Back: P(H | E) = P(E | H) P(H) / P(E).

Front: What is the prior P(H)?
Back: The probability of the hypothesis before seeing the evidence.

Front: What is the likelihood P(E | H)?
Back: How well the hypothesis predicts the observed evidence.

Front: What does the denominator P(E) expand to?
Back: P(E | H) P(H) + P(E | not H) P(not H), by the law of total probability.

Front: Define the posterior P(H | E).
Back: The updated probability of the hypothesis after observing the evidence.

Front: Fraud rate 1%, sensitivity 90%, false positive rate 5%. What is P(fraud | flag)?
Back: About 15.4%: 90 true positives out of 585 total flags in 10,000 transactions.

Front: Why is the fraud posterior so much smaller than the 90% sensitivity?
Back: Clean transactions vastly outnumber fraudulent ones, so false positives swamp true positives.

Front: What is base-rate neglect?
Back: Answering P(H | E) with the likelihood P(E | H) because the prior was ignored.

Front: State the odds form of Bayes' rule.
Back: Posterior odds = prior odds times the likelihood ratio P(E | H) / P(E | not H).

Front: How do you convert odds to a probability?
Back: p = odds / (1 + odds).

Front: In a 2x2 evidence table, what fraction gives the posterior?
Back: The cell of true positives divided by the total of the flagged column.

Front: Does precision behave like a prior, a likelihood or a posterior?
Back: A posterior: P(true | flagged). Recall is the likelihood P(flagged | true).

Front: With prior 0.01 and a test of sensitivity 0.90 and false positive rate 0.05, what happens to the posterior if the prior rises to 0.10?
Back: It rises from about 15% to about 67% with the same test.

Front: Which usually helps more at low prevalence: halving the false positive rate or raising sensitivity from 90% to 95%?
Back: Cutting false positives, because they multiply against a very large pool of negatives.

Front: Why can a 20% posterior still justify acting on an alarm?
Back: When the cost of a missed event far exceeds the cost of a false alarm, expected cost favours acting.

Bayes' rule states that P(H | E) equals the _____ times the prior, divided by the marginal likelihood P(E).

With a 1% base rate, 90% sensitivity and a 5% false positive rate, the posterior probability of fraud given a flag is about _____.

The odds form of Bayes' rule says posterior odds equal prior odds multiplied by the _____.

Answering P(H | E) with P(E | H) because the prior was ignored is called _____.

Dividing the true positives (90) by all flags (585) in the 10,000-transaction table gives the _____ probability of fraud.

To convert odds of 0.182 into a probability, compute p = _____, which gives about 0.154.

## Avoiding the Base-Rate Trap

- **Write the prior before anything else.** In a screening question, sketch the 10,000-row table first and fill the cells second. The table forces the base rate onto the page, and the answer usually falls out on its own.
- **Prefer natural frequencies to percentages.** "900 of 1,000 flagged transactions come from honest customers" is far harder to misread than "false positive rate 5%". Both say the same thing; only one survives a tired reader.
- **Remember that the denominator is every flag, not every fraud.** The single most common error is dividing the 90 true positives by 100 instead of by 585. If you catch yourself computing 90/100, you have written the sensitivity again.
- **Sensitivity and false positives are not interchangeable.** At low prevalence, cutting the false positive rate in half usually moves the posterior more than pushing sensitivity from 90% to 95%, because the false positives multiply against a huge pool of negatives.
- **A modest posterior is not a useless alarm.** A 15% or 20% posterior can still justify action when the cost of a missed event dwarfs the cost of a false alarm. Bayes' rule gives you the probability; the loss function decides what to do with it.
- **Update in odds when you update twice.** Multiply the prior odds by the first likelihood ratio, then by the second. The odds form avoids recomputing the marginal likelihood each time and keeps the arithmetic honest.

A quick self-check before you trust any conditional probability you have just computed: does the answer live in the same event space as the question? If the question asks about transactions and your number came from a fraud-only denominator, the answer is wrong regardless of how clean the arithmetic looks.

## What to Carry Into the Checkpoint

Bayes' rule turns a prior and a likelihood into a posterior, and the denominator P(E) is what makes the update honest: it counts *every* way the evidence could have appeared, not just the way you are hoping for. Base-rate neglect is the failure to let a small prior drag the posterior down, and the 10,000-row frequency table is the cheapest defence against it.

For this lesson, the numbers worth remembering: with a 1% base rate, 90% sensitivity and a 5% false positive rate, the posterior is about 15%; raise the prior to 10% and the same test yields about 67%. The test did not change, the population did.

The next lesson is the module checkpoint, where you will pick a distribution for a count, a measurement or a waiting time, and run a full Bayes update on a dataset question without any of the scaffolding above.

Front: State Bayes' rule for a hypothesis H and evidence E.
Back: P(H | E) = P(E | H) * P(H) / P(E), where P(E) is the marginal likelihood.

Front: What is the base rate in a diagnostic or fraud question?
Back: The prior P(H): the prevalence of the condition in the population before seeing evidence.

Front: Sensitivity P(E | H) tells you what, exactly?
Back: The probability of a positive result given that the condition is present, not the probability that it is present.

Front: What does the false positive rate P(E | not H) contribute to?
Back: The marginal likelihood: it is multiplied by the size of the healthy or honest population.

Front: In a 10,000-transaction table with base rate 1%, sensitivity 90% and false positive rate 5%, how many true positives and false positives appear?
Back: About 90 true positives and about 495 false positives.

Front: What is the posterior probability of fraud given a flag in that table?
Back: 90 / 585, which is about 15.4%.

Front: Why is the posterior so much smaller than the sensitivity of 90%?
Back: Because the 99% honest majority generates a large pool of false positives that swamps the true positives.

Front: What is base-rate neglect?
Back: Answering P(H | E) with P(E | H) because the prior prevalence was ignored.

Front: Write Bayes' rule in odds form.
Back: Posterior odds = prior odds * likelihood ratio, with likelihood ratio P(E | H) / P(E | not H).

Front: How do you turn odds into a probability?
Back: p = odds / (1 + odds). For odds 0.182 this gives about 0.154.

Front: Which quantity does the marginal likelihood P(E) capture?
Back: The total probability of the evidence across all hypotheses: the true-positive and false-positive paths combined.

Front: A model flags 8% of users with precision 20%. Is P(flagged | true) or P(true | flagged) equal to 20%?
Back: A posterior: P(true | flagged). Recall is the likelihood P(flagged | true).

Front: With prior 0.01 and a test of sensitivity 0.90 and false positive rate 0.05, what happens to the posterior if the prior rises to 0.10?
Back: It rises from about 15% to about 67% with the same test.

Front: Which usually helps more at low prevalence: halving the false positive rate or raising sensitivity from 90% to 95%?
Back: Cutting false positives, because they multiply against a very large pool of negatives.

Front: Why can a 20% posterior still justify acting on an alarm?
Back: When the cost of a missed event far exceeds the cost of a false alarm, expected cost favours acting.

Bayes' rule states that P(H | E) equals the _____ times the prior, divided by the marginal likelihood P(E).

With a 1% base rate, 90% sensitivity and a 5% false positive rate, the posterior probability of fraud given a flag is about _____.

The odds form of Bayes' rule says posterior odds equal prior odds multiplied by the _____.

Answering P(H | E) with P(E | H) because the prior was ignored is called _____.

Dividing the true positives (90) by all flags (585) in the 10,000-transaction table gives the _____ probability of fraud.

To convert odds of 0.182 into a probability, compute p = _____, which gives about 0.154.

---

## Contents

Part of [🎲 Mathematics for Machine Learning 3: Probability and Statistics for Data-Driven Models](https://everwise.study/courses/mathematics-for-machine-learning-3-probability-and-statistics-for-data.md)

- **🎲 Probability Foundations for Data**
  - [🧩 Axioms, Conditional Probability and Independence](https://everwise.study/courses/mathematics-for-machine-learning-3-probability-and-statistics-for-data/probability-axioms-conditional-independence.md)
  - [📊 Expectation and Variance on a Real Dataset](https://everwise.study/courses/mathematics-for-machine-learning-3-probability-and-statistics-for-data/expectation-variance-worked-dataset.md) (sign-in required)
  - [🔗 Covariance, Correlation and Dependence](https://everwise.study/courses/mathematics-for-machine-learning-3-probability-and-statistics-for-data/covariance-correlation-dependence.md)
  - [✅ Checkpoint: Probability Foundations](https://everwise.study/courses/mathematics-for-machine-learning-3-probability-and-statistics-for-data/checkpoint-probability-foundations.md)
- **📈 Distributions and Bayes' Rule**
  - [🔢 Bernoulli, Binomial and Poisson Counts](https://everwise.study/courses/mathematics-for-machine-learning-3-probability-and-statistics-for-data/bernoulli-binomial-poisson-count-data.md)
  - [🔔 Gaussian and Exponential Models](https://everwise.study/courses/mathematics-for-machine-learning-3-probability-and-statistics-for-data/gaussian-exponential-continuous-data.md)
  - [🧠 Bayes' Rule and Base Rates](https://everwise.study/courses/mathematics-for-machine-learning-3-probability-and-statistics-for-data/bayes-rule-base-rates.md) — this lesson
  - [🧪 Checkpoint: Distributions and Bayes](https://everwise.study/courses/mathematics-for-machine-learning-3-probability-and-statistics-for-data/checkpoint-distributions-and-bayes.md)
- **🎯 Estimation and Inference**
  - [⚙️ Maximum Likelihood and MAP Estimation](https://everwise.study/courses/mathematics-for-machine-learning-3-probability-and-statistics-for-data/maximum-likelihood-map-estimation.md)
  - [⚖️ Bias-Variance Tradeoff in Practice](https://everwise.study/courses/mathematics-for-machine-learning-3-probability-and-statistics-for-data/bias-variance-tradeoff-practice.md)
  - [📐 Sampling, Confidence Intervals and Hypothesis Tests](https://everwise.study/courses/mathematics-for-machine-learning-3-probability-and-statistics-for-data/sampling-confidence-intervals-hypothesis-tests.md) (premium content)
  - [🏁 Checkpoint: Estimation and Inference](https://everwise.study/courses/mathematics-for-machine-learning-3-probability-and-statistics-for-data/checkpoint-estimation-and-inference.md) (premium content)
- [Course cheatsheet](https://everwise.study/courses/mathematics-for-machine-learning-3-probability-and-statistics-for-data/cheatsheet.md) (premium content)