Maximum Likelihood and MAP Estimation

By Better Wise

Your banner test is live. After 12 impressions you have 3 clicks, and someone asks: "So what's the true click rate?" This lesson answers that question twice, once with maximum likelihood and once with a prior, and shows why the second answer is the one a careful data scientist reports on small samples.

In Module 2 you used Bayes' rule to turn a base rate into a posterior. Here that machinery becomes an estimation tool: the posterior's peak is a point estimate.

By the end you can:

• write a likelihood for a Bernoulli or Gaussian model and maximize it with algebra
• log-transform a likelihood without changing the answer
• derive a MAP estimate for the same data and quantify the prior's pull
• explain shrinkage and say when it fades

The likelihood is the probability of the data you actually saw

Model each impression as $X_i \sim \text{Bernoulli}(p)$. The dataset question becomes "what value of $p$ makes 3 clicks in 12 impressions most probable?"

$$ L(p) = \prod_{i=1}^{n} p^{x_i}(1-p)^{1-x_i} = p^{k}(1-p)^{n-k}, \quad k=3,\ n=12 $$

Two facts to internalize. First, $L(p)$ is a function of the parameter, not of the data: the data are fixed, and you slide $p$ around. Second, the likelihood is not a probability distribution over $p$, so it does not integrate to 1. A likelihood is not a posterior.

Products are awkward, so take logs (monotone, same maximizer):

$$ \ell(p) = k\log p + (n-k)\log(1-p) $$

MLE for a click rate: from product to log to 3/12

Set the derivative to zero. This is the only calculus you need, and it is one line:

$$ \frac{d\ell}{dp} = \frac{k}{p} - \frac{n-k}{1-p} = 0 ;\Rightarrow; \hat{p}_{\text{MLE}} = \frac{k}{n} = \frac{3}{12} = 0.25 $$

The second derivative is negative, so this is a maximum. The MLE is just the observed frequency. For 12 rows that is a fragile number: one extra click moves it to 0.31, one fewer to 0.17. Nothing in the MLE knows that click rates are usually small.

Same recipe, Gaussian mean. With $x_i \sim N(\mu, \sigma^2)$ and $\sigma$ known, $\ell(\mu) = -\frac{1}{2\sigma^2}\sum (x_i-\mu)^2 + \text{const}$, so maximizing the likelihood is minimizing squared error. The MLE of $\mu$ is the sample mean $\bar{x}$. The MLE of $\sigma^2$ is $\frac{1}{n}\sum (x_i - \bar{x})^2$, which divides by $n$ and therefore sits below the unbiased $n-1$ version. That small downward bias is a known cost of pure likelihood.

Adding a prior: MAP and the Beta shrinkage

Suppose past banners on your site clicked at about 0.2. Encode that as $\text{Beta}(2,8)$, which has mean $2/10 = 0.2$ and behaves like 10 pseudo-observations: 2 clicks, 8 non-clicks.

Bayes gives posterior $\propto$ likelihood $\times$ prior, so the MAP estimate is the posterior's peak:

$$ \hat{p}_{\text{MAP}} = \arg\max_p \left[ k\log p + (n-k)\log(1-p) + (a-1)\log p + (b-1)\log(1-p) \right] $$

Combine the log terms and set the derivative to zero:

$$ \hat{p}_{\text{MAP}} = \frac{k + a - 1}{n + a + b - 2} = \frac{3 + 1}{12 + 8} = \frac{4}{20} = 0.20 $$

Notice the exponents: the Beta prior contributes $a-1$ and $b-1$, not $a$ and $b$. That off-by-one is the most common error in this formula. Also notice that the prior mean happened to equal the MAP here only because the data landed near it; the general statement is that MAP is a precision-weighted compromise.

The prior does not win, it negotiates. As $n$ grows, the likelihood term dominates:

Shrinkage is strongest exactly where you need it, on small samples.

MAP for a Gaussian mean: same shrinkage, different algebra

Delivery times are modeled as $N(\mu, \sigma^2)$ with $\sigma = 2$ minutes known, and operations believes $\mu$ is near 6 minutes, so $\mu \sim N(6, 1)$. You observe $n=9$ deliveries with $\bar{x} = 7.2$.

The MLE is $\hat\mu_{\text{MLE}} = 7.2$. For MAP, precision adds: posterior precision $= n/\sigma^2 + 1/\tau^2 = 9/4 + 1 = 3.25$, and

$$ \hat\mu_{\text{MAP}} = \frac{n\bar{x}/\sigma^2 + \mu_0/\tau^2}{n/\sigma^2 + 1/\tau^2} = \frac{16.2 + 6}{3.25} \approx 6.83 $$

Equivalently, it is a weighted average with weight $w = \frac{n\tau^2}{n\tau^2 + \sigma^2} = \frac{9}{13} \approx 0.69$ on the data: $6 + 0.69(1.2) = 6.83$. The belief pulled the estimate 0.37 minutes toward 6, and it stops pulling once $n\tau^2$ dwarfs $\sigma^2$.

Note: MLE is the special case of MAP with a flat prior. When a prior is constant, maximizing likelihood times prior is the same as maximizing likelihood.

Click to reveal a quick check

Same banner, but you only had 4 impressions and 1 click, with the same Beta(2,8) prior. MLE is 0.25; MAP is $(1+1)/(4+8) = 0.167$. The smaller sample gives the prior more relative weight, so the estimate sits closer to the prior mean of 0.2.

Rules of thumb that keep MLE and MAP honest

• Log first, differentiate second. Never differentiate a product of 12 terms by hand.
• Say "likelihood", not "probability of $p$". Reviewers notice.
• Check whether your model's data are counts (Bernoulli, binomial, Poisson) or continuous (Gaussian, exponential). The MLE formula follows from the model, not from habit.
• Report which prior you used and how many pseudo-observations it is worth. A prior is an assumption, and someone will ask.
• Do not use MAP when the question is about uncertainty. MAP gives a point; a confidence interval or the full posterior gives a range, which is next lesson's subject.
• Prefer the unbiased variance $\frac{1}{n-1}\sum (x_i-\bar{x})^2$ when reporting spread, and know that the MLE divides by $n$.

Try it: a Poisson count model has MLE $\hat\lambda = \bar{x}$. If your prior on $\lambda$ is $\text{Gamma}(\alpha, \beta)$, what is the MAP? Work the log-posterior $\ell(\lambda) = (\sum x_i + \alpha - 1)\log\lambda - (n + \beta)\lambda$ and you get $\frac{\sum x_i + \alpha - 1}{n + \beta}$. Same structure, one more prior parameter.

The log-likelihood of k successes in n Bernoulli trials is k log p + (n-k) log(1-p), and its maximizer is _____. A MAP estimate maximizes the _____, which is proportional to the likelihood times the prior. With a Beta(a, b) prior, the MAP click rate is _____. When the prior is _____, the MAP estimate coincides with the MLE. As n grows, the prior's pull _____, and the MAP estimate approaches the MLE.

Front: What does the likelihood L(θ) measure? Back: The probability or density of the observed data, viewed as a function of the parameter θ.

Front: MLE for a Bernoulli click rate with k clicks in n impressions? Back: p̂ = k/n, the observed frequency.

Front: Why take the log of a likelihood? Back: Products become sums, the maximizer is unchanged, and derivatives are simpler.

Front: How is the MAP estimate defined? Back: The value of θ that maximizes likelihood times prior, i.e. the posterior mode.

Front: MAP for a Bernoulli rate with a Beta(a, b) prior? Back: (k + a - 1) / (n + a + b - 2).

Front: Which prior makes MAP identical to MLE? Back: A flat, constant prior.

Front: What does a Beta(2, 8) prior on a click rate encode? Back: A prior mean of 0.2, worth about 10 pseudo-observations (2 clicks, 8 non-clicks).

Front: How does the prior's influence change as n grows? Back: It shrinks, because the likelihood term grows with n and MAP converges to MLE.

Front: MLE of a Gaussian mean with known variance? Back: The sample mean x̄.

Front: MAP of a Gaussian mean with prior N(mu0, tau^2) and known sigma^2? Back: A precision-weighted average: (n x̄/sigma^2 + mu0/tau^2) / (n/sigma^2 + 1/tau^2).

Front: MLE of a Gaussian variance, and its flaw? Back: (1/n) Σ (xi - x̄)^2, which is biased low because it divides by n rather than n-1.

Front: In the banner example with n=12, k=3 and a Beta(2,8) prior, what are MLE and MAP? Back: MLE = 0.25, MAP = 4/20 = 0.20.

Front: How does MAP differ from a full Bayesian posterior? Back: MAP is a single point (the mode); the posterior keeps the whole distribution and its uncertainty.

What to carry into the bias-variance lesson

You now have two estimators for one number: $\hat{p}{\text{MLE}} = 0.25$ and $\hat{p}{\text{MAP}} = 0.20$, differing only by an assumption. Likelihood answers "what do these rows say", a prior answers "what else do I believe", and MAP blends them with a weight that depends on sample size.

Next, in Bias-Variance Tradeoff in Practice, we ask which of these two estimators you should actually deploy, and how that choice shows up as error you can measure.