20 · Point estimation, confidence intervals, and MLE
Maximum Likelihood Estimation gives you the best-guess parameter; a confidence interval tells you how uncertain that guess is — both rooted in the CLT.
A point estimate is your best guess; a confidence interval quantifies the uncertainty; MLE is the principle behind both.
Without this:
Without estimation, every model output would be a number with no uncertainty attached — useless for decisions.
Point estimation is the process of using sample data to produce a single value — a point estimate — that serves as a best guess of an unknown population parameter.
Key vocabulary:
- Parameter (θ) — a fixed but unknown property of the population (e.g. the true mean μ, or the true proportion p).
- Estimator (θ̂) — a function of the sample data that we use to estimate θ. The sample mean X̄ is an estimator of μ.
- Estimate — the specific numerical value of θ̂ for a given sample.
Properties of good estimators:
- Unbiasedness: E[θ̂] = θ. The estimator is correct on average. Sample mean is unbiased for μ; sample variance with n−1 (not n) is unbiased for σ².
- Consistency: As n → ∞, θ̂ → θ in probability. More data brings you closer to the truth.
- Efficiency: Among all unbiased estimators, the one with the smallest variance is most efficient.
Maximum Likelihood Estimation (MLE): The MLE finds the parameter value θ̂ that makes the observed data most probable. If the data x₁, …, xₙ are iid from f(x | θ), the log-likelihood is:
ℓ(θ) = Σᵢ log f(xᵢ | θ)
and θ̂_MLE = arg max_θ ℓ(θ).
For the Normal distribution, setting ∂ℓ/∂μ = 0 gives θ̂_MLE = x̄ — the sample mean.
Confidence intervals: A 95% confidence interval for the mean (when σ is known) is: x̄ ± 1.96 · σ/√n
Interpretation: if you repeated the experiment many times and computed this interval each time, 95% of those intervals would contain the true μ. It is NOT "95% probability that μ is in this interval" — μ is fixed, not random.
When σ is unknown, replace σ with the sample standard deviation s and replace z with the t-quantile (t-distribution, covered next lesson).
Bootstrap CI (model-free): When no closed-form formula exists, resample from the data with replacement many times (10,000+), compute the statistic each time, and take the 2.5th and 97.5th percentiles as the 95% CI.
Python (in browser)
MLE for the Normal mean is simply the sample mean — obtained by solving ∂ℓ/∂μ = 0 analytically. scipy.stats.norm.fit returns the same MLE values. Note the small negative bias in the MLE σ̂ (ddof=0); using ddof=1 corrects it.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
The 95% CI is x̄ ± 1.96·σ/√n. Running 1000 independent experiments and checking whether each CI captures the true μ = 50 confirms the coverage is indeed ≈ 95%. This is the frequentist meaning of a confidence interval.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
Bootstrap CI: resample the data with replacement 10,000 times, compute the mean each time, take the 2.5th and 97.5th percentiles. No distributional assumptions needed — it works for any statistic (mean, median, AUC, F1, …).
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Maximising the Normal log-likelihood with respect to μ gives μ̂ = x̄ — the sample mean. For σ², the MLE is the biased (1/n) formula; the unbiased version uses n−1. This derivation is the algebraic link between least squares and the Gaussian assumption.
What does 'unbiased estimator' mean?
The MLE derivation and its connection to MAP estimation (MLE + prior = MAP) are worked out in detail in MML lesson 43. Understanding the log-likelihood calculus there will let you derive the MLE for any distribution — not just the Normal.
- **MLE:** find θ that maximises ℓ(θ) = Σ log f(xᵢ | θ). For Normal data, MLE gives μ̂ = x̄.
- **95% CI:** x̄ ± 1.96·σ/√n. Coverage interpretation: 95% of such intervals (across repeated experiments) contain the true μ.
- **Bootstrap CI:** resample with replacement → compute statistic → take 2.5th & 97.5th percentiles. Works for any statistic.
Every parameter in every supervised model is an MLE (under Gaussian residuals = least squares; under Bernoulli labels = logistic regression). Bootstrap CIs are how you put error bars on F1, AUC, RMSE.
If you remove it: You can't justify why minimizing MSE is the 'right' loss for linear regression — it's the MLE under Gaussian residuals.