Parameter Estimation (MLE and MAP)

Probabilistic models recast learning as parameter estimation. Assume the data is generated by $p(y \mid \mathbf{x}, \theta)$ — a family of distributions indexed by $\theta$. Given i.i.d. samples $\mathcal{D}$, the likelihood is $p(\mathcal{D} \mid \theta) = \prod_n p(y_n \mid \mathbf{x}_n, \theta)$. It is the probability the model assigns to the data as a function of the parameters.

Maximum Likelihood Estimation (MLE) picks the parameters that make the observed data most probable: We work with the log-likelihood because it turns products into sums (numerically stable) and because the $\log$ is monotonic (same argmax). For a Gaussian observation model with fixed variance, maximizing the log-likelihood is exactly minimizing squared error — a beautiful unification of probabilistic and ERM viewpoints.

Maximum a Posteriori (MAP) estimation folds in a prior $p(\theta)$. By Bayes' rule, the posterior is $p(\theta \mid \mathcal{D}) \propto p(\mathcal{D} \mid \theta)\, p(\theta)$, and MAP maximizes this: The prior acts as a regularizer. A Gaussian prior $\theta \sim \mathcal{N}(\mathbf{0}, \tau^2 I)$ gives an $\ell_2$ penalty (Ridge regression). A Laplace prior gives $\ell_1$ (Lasso). Regularization is not a hack — it is a principled Bayesian prior expressed through the posterior mode.

MLE is consistent and asymptotically efficient under regularity conditions: as $N \to \infty$, $\hat\theta_{\text{MLE}} \to \theta^*$ and its variance achieves the Cramér-Rao lower bound. MAP inherits these properties when the prior is informative but not overwhelming. When the prior dominates ($N$ small), MAP is biased toward the prior; when likelihood dominates ($N$ large), MAP and MLE agree.

Both MLE and MAP return point estimates — a single $\hat\theta$. They ignore uncertainty in $\theta$ itself, which can lead to overconfident predictions. Fully Bayesian inference, which we'll explore in linear regression, keeps the entire posterior $p(\theta \mid \mathcal{D})$ and propagates uncertainty into predictions: $p(y_* \mid \mathbf{x}_*, \mathcal{D}) = \int p(y_* \mid \mathbf{x}_*, \theta)\, p(\theta \mid \mathcal{D})\, d\theta$.

Worked example — MLE for coin bias: Given 7 heads in 10 tosses, the likelihood is $p(\mathcal{D} \mid p) = p^7 (1-p)^3$. Taking $\log$ and differentiating: $\frac{7}{p} - \frac{3}{1-p} = 0 \Rightarrow 7(1-p) = 3p \Rightarrow p = 0.7$. The MLE simply matches the empirical frequency — as it should for a Bernoulli likelihood.

Worked example — MAP with Beta(2,2) prior on same coin: With prior $p(\theta) \propto \theta(1-\theta)$, the posterior is $\theta^{7+1}(1-\theta)^{3+1} = \theta^8(1-\theta)^4$. Maximizing: $\frac{8}{\theta} - \frac{4}{1-\theta} = 0 \Rightarrow \theta = 8/12 \approx 0.667$. The prior shrinks the estimate from 0.7 toward 0.5 — less aggressive than MLE, exactly the regularization effect of a non-flat prior.