Conjugacy and Exponential Family

In Bayesian inference, we update a prior $p(\theta)$ with a likelihood $p(\mathcal{D} \mid \theta)$ to get a posterior $p(\theta \mid \mathcal{D}) \propto p(\mathcal{D} \mid \theta)p(\theta)$. The posterior is generally messy — but for certain conjugate prior-likelihood pairs, the posterior has the same functional form as the prior. This turns inference from integral calculus into parameter arithmetic.

Classical conjugate pairs: Beta prior + Bernoulli/Binomial likelihood → Beta posterior; Gaussian prior (known variance) + Gaussian likelihood → Gaussian posterior; Dirichlet prior + Multinomial → Dirichlet posterior; Gamma prior + Poisson → Gamma. The math reduces to updating 'effective counts' or 'natural parameters', avoiding integration entirely.

The deeper reason these exist: the exponential family of distributions. A family has exponential-family form if $p(x \mid \theta) = h(x) \exp\left(\boldsymbol\eta(\theta)^\top T(x) - A(\theta)\right)$, where $T(x)$ are the sufficient statistics, $\boldsymbol\eta$ are natural parameters, and $A$ is the log-partition function. Gaussian, Bernoulli, Poisson, Beta, Dirichlet, Gamma, and categorical distributions are all exponential families.

Exponential families have beautiful properties. Sufficient statistics carry all information from the data about $\theta$ — you can throw away the raw data. The MLE comes from matching the expected sufficient statistics to the empirical ones. Any exponential family has a conjugate prior of a specific form. And the family is closed under multiplication (so posteriors stay in it).

In ML, this framework clarifies many methods. Logistic regression and Poisson regression are exponential families with the linear predictor feeding the natural parameter — hence generalized linear models (GLMs). Variational inference with exponential family approximations becomes coordinate-wise parameter updates. Even modern energy-based models are unnormalized exponential families. Seeing the family lurking behind a model is a shortcut to both theory and efficient algorithms.

Worked example — Beta-Binomial conjugacy: Start with prior $\text{Beta}(2,2)$ (weak belief centered at 0.5). Observe 7 heads in 10 flips. The posterior is $\text{Beta}(2+7,\; 2+3) = \text{Beta}(9, 5)$. Its mean is $9/(9+5) \approx 0.643$ — shifted from the prior mean of 0.5 toward the data frequency 0.7, but pulled back a little by the prior.

Worked example — exponential-family form of Bernoulli: $p(x \mid \theta) = \theta^x(1-\theta)^{1-x}$ rewrites as $\exp\left(x \log\frac{\theta}{1-\theta} + \log(1-\theta)\right)$. So the natural parameter is $\eta = \log\frac{\theta}{1-\theta}$ (the logit), the sufficient statistic is $T(x) = x$, and $A(\theta) = -\log(1-\theta)$. This is exactly the logit used in logistic regression.