Summary Statistics and Independence

Distributions are high-dimensional objects, but we often compress them into a few summary statistics. The expected value (mean) $\mathbb{E}[X] = \sum_x x\, p(x)$ (or $\int x\, p(x)\, dx$) gives the long-run average. Expectation is linear: $\mathbb{E}[aX + bY + c] = a\mathbb{E}[X] + b\mathbb{E}[Y] + c$ — no independence required. This linearity is the single most-used identity in probabilistic ML.

The variance $\text{Var}[X] = \mathbb{E}[(X - \mathbb{E}[X])^2]$ measures spread around the mean. Its square root $\sigma = \sqrt{\text{Var}[X]}$ is the standard deviation, measured in the same units as $X$. Variance is not linear but satisfies $\text{Var}[aX + b] = a^2 \text{Var}[X]$ — constants shift the mean but not the spread.

For two random variables, the covariance $\text{Cov}[X, Y] = \mathbb{E}[(X - \mathbb{E} X)(Y - \mathbb{E} Y)]$ measures linear co-variation. Positive means they tend to move together; negative means oppositely; zero means no linear relationship. The normalized version is the correlation $\rho = \text{Cov}[X, Y] / (\sigma_X \sigma_Y) \in [-1, 1]$.

For multiple random variables bundled as a vector $\mathbf{X}$, the covariance matrix $\Sigma$ has entries $\Sigma_{ij} = \text{Cov}[X_i, X_j]$. Diagonal entries are variances; off-diagonals are covariances. Covariance matrices are always symmetric positive semi-definite — a fact exploited heavily by PCA, Gaussian processes, and variational methods.

Independence implies zero covariance, but not conversely: zero correlation does not imply independence. Two variables can be perfectly dependent with zero correlation (e.g., $Y = X^2$ with $X \sim \mathcal{N}(0, 1)$). This is a constant source of bugs: if your model assumes independence based on zero correlation, you may miss crucial nonlinear relationships. In ML, we usually check with mutual information or other nonlinear measures when independence is a critical assumption.