GMM: Latent Variable Perspective

The cleanest way to derive GMMs and EM is via latent variables. Introduce a categorical latent $z_n \in \{1, \ldots, K\}$ for each data point, with prior $p(z_n = k) = \pi_k$. Conditional on $z_n = k$, the observation is Gaussian: $p(\mathbf{x}_n \mid z_n = k) = \mathcal{N}(\mathbf{x}_n; \boldsymbol\mu_k, \Sigma_k)$. Marginalizing out $z_n$ recovers the mixture density $p(\mathbf{x}_n) = \sum_k \pi_k \mathcal{N}(\mathbf{x}_n; \boldsymbol\mu_k, \Sigma_k)$.

The responsibility $r_{nk}$ from the E-step is precisely the posterior of the latent: This is Bayes' rule for the hidden cluster indicator. The EM algorithm now has a clean interpretation: compute the posterior over hidden variables (E), then maximize the expected complete-data log-likelihood $\mathbb{E}_{p(z \mid \mathbf{x}, \boldsymbol\theta^{\text{old}})}[\log p(\mathbf{x}, z \mid \boldsymbol\theta)]$ (M).

The complete-data log-likelihood is $\sum_n \sum_k \mathbb{1}[z_n = k] \left[ \log \pi_k + \log \mathcal{N}(\mathbf{x}_n; \boldsymbol\mu_k, \Sigma_k) \right]$. Taking expectations with respect to the responsibilities replaces the indicators with $r_{nk}$, producing a sum of weighted log-Gaussians — decoupled and easy to maximize. This decoupling is why M-step closed-forms exist: conditional on knowing the soft assignments, the problem factorizes.

Once trained, responsibilities reveal soft cluster membership. A point near a cluster center has $r_{nk} \approx 1$ for that $k$; a point equidistant from two clusters splits mass. This is richer than $k$-means' hard assignment — the model knows when it is uncertain. We can use responsibilities to detect outliers (points with low responsibility for every cluster have low $p(\mathbf{x}_n)$) or to visualize uncertainty in cluster boundaries.

This latent-variable framework generalizes far beyond GMMs. Hidden Markov Models use a sequence of latent states. Factor Analysis and Probabilistic PCA have continuous latents. Variational Autoencoders learn deep latent representations with neural-network likelihoods. Latent Dirichlet Allocation assigns topics to documents. Every time you see EM, variational inference, or sampling — there is a latent variable structure underneath, and the GMM is the canonical worked example.

Worked example — unequal priors reshape responsibility: Same two components $\mathcal{N}(-1, 1)$ and $\mathcal{N}(1, 1)$ but now $\pi_1 = 0.9, \pi_2 = 0.1$ (cluster 1 is much more common). For $x = 0.5$ we previously got $r_2 \approx 0.73$. Now the joint numerators are $0.9 e^{-1.125}$ and $0.1 e^{-0.125}$. Ratio: $r_2 / r_1 = (0.1 / 0.9) \cdot e^{1} \approx 0.302$, so $r_2 \approx 0.232$ and $r_1 \approx 0.768$. The strong prior flipped the assignment — Bayes' rule in action.

Worked example — complete-data vs observed-data log-likelihood: Suppose one point $x = 0$ is assigned responsibility $r_1 = 0.7, r_2 = 0.3$ under two components $\mathcal{N}(-1, 1), \mathcal{N}(1, 1)$ with equal priors. Observed log-likelihood: $\log(0.5 \mathcal{N}(0;-1,1) + 0.5\mathcal{N}(0;1,1)) = \log(\mathcal{N}(0;-1,1)) \approx -1.42$ (plus a constant). Expected complete-data log-likelihood: $0.7 [\log 0.5 + \log\mathcal{N}(0;-1,1)] + 0.3 [\log 0.5 + \log\mathcal{N}(0;1,1)]$, cleanly decoupled across clusters — M-step can maximize each component independently.