Bayesian Linear Regression

Bayesian linear regression treats the weights $\boldsymbol\beta$ as a *random variable* rather than an unknown constant. We start with a prior $p(\boldsymbol\beta) = \mathcal{N}(\mathbf{0}, \Sigma_0)$, observe data, and apply Bayes' rule to obtain the posterior $p(\boldsymbol\beta \mid \mathcal{D})$. The payoff: instead of a single point estimate, we get a full distribution over weights that quantifies uncertainty.

Conjugacy is our friend. With Gaussian likelihood $p(\mathbf{y} \mid X, \boldsymbol\beta) = \mathcal{N}(\mathbf{y}; X\boldsymbol\beta, \sigma^2 I)$ and Gaussian prior $\mathcal{N}(\mathbf{0}, \tau^2 I)$, the posterior is also Gaussian: where $\Sigma_N = \left( \frac{1}{\sigma^2} X^\top X + \frac{1}{\tau^2} I \right)^{-1}$ and $\boldsymbol\mu_N = \frac{1}{\sigma^2} \Sigma_N X^\top \mathbf{y}$. The posterior mean $\boldsymbol\mu_N$ equals the ridge regression solution — MAP is the posterior mode, which for Gaussians is also the mean.

The real prize is the posterior predictive distribution for a new point $\mathbf{x}_*$: The predictive variance has two parts: parameter uncertainty ($\mathbf{x}_*^\top \Sigma_N \mathbf{x}_*$, which shrinks with more data) and irreducible noise ($\sigma^2$). This is what gives Bayesian regression its error bars — predictions in well-sampled regions are confident; predictions far from training data carry large uncertainty.

Bayesian regression degrades gracefully. With zero data the prior dominates: $\boldsymbol\mu_N = \mathbf{0}$, $\Sigma_N = \tau^2 I$, and the predictive distribution is wide everywhere. As $N \to \infty$, the posterior concentrates at the true $\boldsymbol\beta$ and the predictive variance approaches $\sigma^2$. This Bayesian updating is a smooth interpolation between prior knowledge and data-driven fit, with uncertainty tracking at every step.

The connection to Gaussian processes is close: GPs are essentially Bayesian linear regression with infinitely many features, parameterized by a kernel instead of a finite feature map. Modern deep learning borrows the intuition: Bayesian neural networks, MC dropout, and deep ensembles are all attempts to recover posterior predictive uncertainty for richer models. The theory never goes away — it only gets harder to implement exactly.

Worked example — posterior contracts with more data: Add a second observation $y_2 = 2.2$ at $x_2 = 1$. The posterior precision becomes $1 + 4 + 4 = 9$, so $\Sigma_N = 1/9 \approx 0.11$. Posterior mean is $(1/9) \cdot 4 \cdot (1 \cdot 2 + 1 \cdot 2.2) = 4.2 \cdot 4 / 9 \approx 1.87$. Variance halved and the estimate moved closer to the empirical average $2.1$ — prior influence fades as data accumulates.