# ⚙️ Maximum Likelihood and MAP Estimation

By Better Wise

Your banner test is live. After 12 impressions you have 3 clicks, and someone asks: "So what's the true click rate?" This lesson answers that question twice, once with maximum likelihood and once with a prior, and shows why the second answer is the one a careful data scientist reports on small samples.

In Module 2 you used Bayes' rule to turn a base rate into a posterior. Here that machinery becomes an estimation tool: the posterior's peak is a point estimate.

By the end you can:
- write a likelihood for a Bernoulli or Gaussian model and maximize it with algebra
- log-transform a likelihood without changing the answer
- derive a MAP estimate for the same data and quantify the prior's pull
- explain shrinkage and say when it fades

## The likelihood is the probability of the data you actually saw

Model each impression as $X_i \sim \text{Bernoulli}(p)$. The dataset question becomes "what value of $p$ makes 3 clicks in 12 impressions most probable?"

$$
L(p) = \prod_{i=1}^{n} p^{x_i}(1-p)^{1-x_i} = p^{k}(1-p)^{n-k}, \quad k=3,\ n=12
$$

Two facts to internalize. First, $L(p)$ is a function of the parameter, not of the data: the data are fixed, and you slide $p$ around. Second, the likelihood is not a probability distribution over $p$, so it does not integrate to 1. **A likelihood is not a posterior.**

Products are awkward, so take logs (monotone, same maximizer):

$$
\ell(p) = k\log p + (n-k)\log(1-p)
$$

## MLE for a click rate: from product to log to 3/12

Set the derivative to zero. This is the only calculus you need, and it is one line:

$$
\frac{d\ell}{dp} = \frac{k}{p} - \frac{n-k}{1-p} = 0 \;\Rightarrow\; \hat{p}_{\text{MLE}} = \frac{k}{n} = \frac{3}{12} = 0.25
$$

The second derivative is negative, so this is a maximum. The MLE is just the observed frequency. For 12 rows that is a fragile number: one extra click moves it to 0.31, one fewer to 0.17. Nothing in the MLE knows that click rates are usually small.

**Same recipe, Gaussian mean.** With $x_i \sim N(\mu, \sigma^2)$ and $\sigma$ known, $\ell(\mu) = -\frac{1}{2\sigma^2}\sum (x_i-\mu)^2 + \text{const}$, so maximizing the likelihood is minimizing squared error. The MLE of $\mu$ is the sample mean $\bar{x}$. The MLE of $\sigma^2$ is $\frac{1}{n}\sum (x_i - \bar{x})^2$, which divides by $n$ and therefore sits below the unbiased $n-1$ version. That small downward bias is a known cost of pure likelihood.

## Adding a prior: MAP and the Beta shrinkage

Suppose past banners on your site clicked at about 0.2. Encode that as $\text{Beta}(2,8)$, which has mean $2/10 = 0.2$ and behaves like 10 pseudo-observations: 2 clicks, 8 non-clicks.

Bayes gives posterior $\propto$ likelihood $\times$ prior, so the MAP estimate is the posterior's peak:

$$
\hat{p}_{\text{MAP}} = \arg\max_p \left[ k\log p + (n-k)\log(1-p) + (a-1)\log p + (b-1)\log(1-p) \right]
$$

Combine the log terms and set the derivative to zero:

$$
\hat{p}_{\text{MAP}} = \frac{k + a - 1}{n + a + b - 2} = \frac{3 + 1}{12 + 8} = \frac{4}{20} = 0.20
$$

Notice the exponents: the Beta prior contributes $a-1$ and $b-1$, not $a$ and $b$. That off-by-one is the most common error in this formula. Also notice that the prior mean happened to equal the MAP here only because the data landed near it; the general statement is that MAP is a precision-weighted compromise.

The prior does not win, it negotiates. As $n$ grows, the likelihood term dominates:

| Data | MLE | MAP, Beta(2,8) prior |
| --- | --- | --- |
| $k=3,\ n=12$ | 0.250 | 0.200 |
| $k=30,\ n=120$ | 0.250 | 0.242 |
| $k=250,\ n=1000$ | 0.250 | 0.249 |

**Shrinkage is strongest exactly where you need it, on small samples.**

## MAP for a Gaussian mean: same shrinkage, different algebra

Delivery times are modeled as $N(\mu, \sigma^2)$ with $\sigma = 2$ minutes known, and operations believes $\mu$ is near 6 minutes, so $\mu \sim N(6, 1)$. You observe $n=9$ deliveries with $\bar{x} = 7.2$.

The MLE is $\hat\mu_{\text{MLE}} = 7.2$. For MAP, precision adds: posterior precision $= n/\sigma^2 + 1/\tau^2 = 9/4 + 1 = 3.25$, and

$$
\hat\mu_{\text{MAP}} = \frac{n\bar{x}/\sigma^2 + \mu_0/\tau^2}{n/\sigma^2 + 1/\tau^2} = \frac{16.2 + 6}{3.25} \approx 6.83
$$

Equivalently, it is a weighted average with weight $w = \frac{n\tau^2}{n\tau^2 + \sigma^2} = \frac{9}{13} \approx 0.69$ on the data: $6 + 0.69(1.2) = 6.83$. The belief pulled the estimate 0.37 minutes toward 6, and it stops pulling once $n\tau^2$ dwarfs $\sigma^2$.

> **Note:** 
> MLE is the special case of MAP with a flat prior. When a prior is constant, maximizing likelihood times prior is the same as maximizing likelihood.

**Click to reveal a quick check**

Same banner, but you only had 4 impressions and 1 click, with the same Beta(2,8) prior. MLE is 0.25; MAP is $(1+1)/(4+8) = 0.167$. The smaller sample gives the prior more relative weight, so the estimate sits closer to the prior mean of 0.2.

## Rules of thumb that keep MLE and MAP honest

- Log first, differentiate second. Never differentiate a product of 12 terms by hand.
- Say "likelihood", not "probability of $p$". Reviewers notice.
- Check whether your model's data are counts (Bernoulli, binomial, Poisson) or continuous (Gaussian, exponential). The MLE formula follows from the model, not from habit.
- Report which prior you used and how many pseudo-observations it is worth. A prior is an assumption, and someone will ask.
- Do not use MAP when the question is about uncertainty. MAP gives a point; a confidence interval or the full posterior gives a range, which is next lesson's subject.
- Prefer the unbiased variance $\frac{1}{n-1}\sum (x_i-\bar{x})^2$ when reporting spread, and know that the MLE divides by $n$.

Try it: a Poisson count model has MLE $\hat\lambda = \bar{x}$. If your prior on $\lambda$ is $\text{Gamma}(\alpha, \beta)$, what is the MAP? Work the log-posterior $\ell(\lambda) = (\sum x_i + \alpha - 1)\log\lambda - (n + \beta)\lambda$ and you get $\frac{\sum x_i + \alpha - 1}{n + \beta}$. Same structure, one more prior parameter.

The log-likelihood of k successes in n Bernoulli trials is k log p + (n-k) log(1-p), and its maximizer is _____.
A MAP estimate maximizes the _____, which is proportional to the likelihood times the prior.
With a Beta(a, b) prior, the MAP click rate is _____.
When the prior is _____, the MAP estimate coincides with the MLE.
As n grows, the prior's pull _____, and the MAP estimate approaches the MLE.

Front: What does the likelihood L(θ) measure?
Back: The probability or density of the observed data, viewed as a function of the parameter θ.

Front: MLE for a Bernoulli click rate with k clicks in n impressions?
Back: p̂ = k/n, the observed frequency.

Front: Why take the log of a likelihood?
Back: Products become sums, the maximizer is unchanged, and derivatives are simpler.

Front: How is the MAP estimate defined?
Back: The value of θ that maximizes likelihood times prior, i.e. the posterior mode.

Front: MAP for a Bernoulli rate with a Beta(a, b) prior?
Back: (k + a - 1) / (n + a + b - 2).

Front: Which prior makes MAP identical to MLE?
Back: A flat, constant prior.

Front: What does a Beta(2, 8) prior on a click rate encode?
Back: A prior mean of 0.2, worth about 10 pseudo-observations (2 clicks, 8 non-clicks).

Front: How does the prior's influence change as n grows?
Back: It shrinks, because the likelihood term grows with n and MAP converges to MLE.

Front: MLE of a Gaussian mean with known variance?
Back: The sample mean x̄.

Front: MAP of a Gaussian mean with prior N(mu0, tau^2) and known sigma^2?
Back: A precision-weighted average: (n x̄/sigma^2 + mu0/tau^2) / (n/sigma^2 + 1/tau^2).

Front: MLE of a Gaussian variance, and its flaw?
Back: (1/n) Σ (xi - x̄)^2, which is biased low because it divides by n rather than n-1.

Front: In the banner example with n=12, k=3 and a Beta(2,8) prior, what are MLE and MAP?
Back: MLE = 0.25, MAP = 4/20 = 0.20.

Front: How does MAP differ from a full Bayesian posterior?
Back: MAP is a single point (the mode); the posterior keeps the whole distribution and its uncertainty.

## What to carry into the bias-variance lesson

You now have two estimators for one number: $\hat{p}_{\text{MLE}} = 0.25$ and $\hat{p}_{\text{MAP}} = 0.20$, differing only by an assumption. Likelihood answers "what do these rows say", a prior answers "what else do I believe", and MAP blends them with a weight that depends on sample size.

Next, in Bias-Variance Tradeoff in Practice, we ask which of these two estimators you should actually deploy, and how that choice shows up as error you can measure.

---

## Contents

Part of [🎲 Mathematics for Machine Learning 3: Probability and Statistics for Data-Driven Models](https://everwise.study/courses/mathematics-for-machine-learning-3-probability-and-statistics-for-data.md)

- **🎲 Probability Foundations for Data**
  - [🧩 Axioms, Conditional Probability and Independence](https://everwise.study/courses/mathematics-for-machine-learning-3-probability-and-statistics-for-data/probability-axioms-conditional-independence.md)
  - [📊 Expectation and Variance on a Real Dataset](https://everwise.study/courses/mathematics-for-machine-learning-3-probability-and-statistics-for-data/expectation-variance-worked-dataset.md) (sign-in required)
  - [🔗 Covariance, Correlation and Dependence](https://everwise.study/courses/mathematics-for-machine-learning-3-probability-and-statistics-for-data/covariance-correlation-dependence.md)
  - [✅ Checkpoint: Probability Foundations](https://everwise.study/courses/mathematics-for-machine-learning-3-probability-and-statistics-for-data/checkpoint-probability-foundations.md)
- **📈 Distributions and Bayes' Rule**
  - [🔢 Bernoulli, Binomial and Poisson Counts](https://everwise.study/courses/mathematics-for-machine-learning-3-probability-and-statistics-for-data/bernoulli-binomial-poisson-count-data.md)
  - [🔔 Gaussian and Exponential Models](https://everwise.study/courses/mathematics-for-machine-learning-3-probability-and-statistics-for-data/gaussian-exponential-continuous-data.md)
  - [🧠 Bayes' Rule and Base Rates](https://everwise.study/courses/mathematics-for-machine-learning-3-probability-and-statistics-for-data/bayes-rule-base-rates.md)
  - [🧪 Checkpoint: Distributions and Bayes](https://everwise.study/courses/mathematics-for-machine-learning-3-probability-and-statistics-for-data/checkpoint-distributions-and-bayes.md)
- **🎯 Estimation and Inference**
  - [⚙️ Maximum Likelihood and MAP Estimation](https://everwise.study/courses/mathematics-for-machine-learning-3-probability-and-statistics-for-data/maximum-likelihood-map-estimation.md) — this lesson
  - [⚖️ Bias-Variance Tradeoff in Practice](https://everwise.study/courses/mathematics-for-machine-learning-3-probability-and-statistics-for-data/bias-variance-tradeoff-practice.md)
  - [📐 Sampling, Confidence Intervals and Hypothesis Tests](https://everwise.study/courses/mathematics-for-machine-learning-3-probability-and-statistics-for-data/sampling-confidence-intervals-hypothesis-tests.md) (premium content)
  - [🏁 Checkpoint: Estimation and Inference](https://everwise.study/courses/mathematics-for-machine-learning-3-probability-and-statistics-for-data/checkpoint-estimation-and-inference.md) (premium content)
- [Course cheatsheet](https://everwise.study/courses/mathematics-for-machine-learning-3-probability-and-statistics-for-data/cheatsheet.md) (premium content)