Probabilistic Modeling and Model Selection

Once you accept a probabilistic view, *every* modeling decision — the functional form, the noise distribution, the prior — is a modeling assumption that can be compared against alternatives. Model selection is the task of choosing among candidate models $\mathcal{M}_1, \mathcal{M}_2, \ldots$ based on how well each explains the data without overfitting.

The Bayesian answer is the marginal likelihood (a.k.a. evidence): The evidence automatically embodies Occam's razor: complex models have broader prior, spreading probability mass over a larger parameter space, so they only 'win' when the data really demands the extra flexibility. The Bayes factor $p(\mathcal{D} \mid \mathcal{M}_1) / p(\mathcal{D} \mid \mathcal{M}_2)$ compares two models directly.

Exact evidence is usually intractable, so practitioners use information criteria as computable approximations. The Bayesian Information Criterion is $\text{BIC} = -2 \log p(\mathcal{D} \mid \hat\theta_{\text{MLE}}) + k \log N$, where $k$ is the number of parameters and $N$ is the sample size; it is a large-$N$ approximation to $-2 \log p(\mathcal{D} \mid \mathcal{M})$. The Akaike Information Criterion is $\text{AIC} = -2 \log p(\mathcal{D} \mid \hat\theta_{\text{MLE}}) + 2k$; it estimates expected out-of-sample loss. Both prefer lower values. BIC penalizes complexity more aggressively than AIC.

In practice, cross-validation (especially $k$-fold) is the frequentist workhorse: partition the data, train on $k-1$ folds, validate on the remaining fold, average. Unlike AIC/BIC, CV makes no distributional assumptions and directly estimates generalization error. The tradeoff is computation: each hyperparameter configuration requires $k$ model fits.

Finally, Bayesian model averaging (BMA) avoids picking at all: it weights predictions by posterior model probability, $p(y_* \mid \mathbf{x}_*, \mathcal{D}) = \sum_m p(y_* \mid \mathbf{x}_*, \mathcal{M}_m)\, p(\mathcal{M}_m \mid \mathcal{D})$. BMA is the principled way to hedge uncertainty across a model family, and in practice ensembles (random forests, bagged networks) are approximate BMA. Model selection vs. averaging is a philosophical choice: do you commit to one story, or keep a portfolio of compatible ones?

Worked example — BIC: linear vs quadratic on $N = 50$: Suppose a linear fit ($k = 2$) yields log-likelihood $-120$, while a quadratic fit ($k = 3$) reaches $-115$. $\text{BIC}_{\text{lin}} = 240 + 2\log 50 \approx 247.82$; $\text{BIC}_{\text{quad}} = 230 + 3\log 50 \approx 241.74$. Quadratic wins by ~6 — modest evidence that the extra parameter is justified. With only $N = 10$, the penalty gap would be smaller and the verdict could flip.

Worked example — AIC vs BIC disagreement: Same setup with $N = 1000$: $\text{AIC}_{\text{lin}} = 244, \text{AIC}_{\text{quad}} = 236$ (quadratic by 8); $\text{BIC}_{\text{lin}} = 253.8, \text{BIC}_{\text{quad}} = 250.7$ (quadratic by 3). BIC's heavier $\log N$ penalty makes it more conservative — when AIC and BIC disagree, it usually signals a borderline complexity call worth investigating with cross-validation.