Change of Variables

Given a random variable $X$ with density $p_X$, what is the density of $Y = f(X)$? The answer is the change of variables formula: for a smooth, invertible $f$, $p_Y(y) = p_X(f^{-1}(y)) \left| \frac{df^{-1}}{dy} \right|$. The Jacobian factor accounts for how $f$ stretches or compresses probability density — concentrations of probability move with the inverse stretch.

In higher dimensions, this becomes $p_Y(\mathbf{y}) = p_X(f^{-1}(\mathbf{y})) \left| \det J_{f^{-1}}(\mathbf{y}) \right|$, where $J_{f^{-1}}$ is the Jacobian matrix of the inverse. Equivalently, using the forward Jacobian: $p_Y(\mathbf{y}) = p_X(\mathbf{x}) / |\det J_f(\mathbf{x})|$ where $\mathbf{x} = f^{-1}(\mathbf{y})$. Either form says: volume scaling of $f$ must be corrected so total probability stays 1.

The formula requires $f$ to be a bijection (one-to-one and onto on the relevant domain) and to be smooth enough for its Jacobian to exist. If $f$ is not invertible — think $Y = X^2$ for $X \in \mathbb{R}$ — you sum contributions from all preimages. If $f$ collapses dimensions (e.g., a rank-deficient transformation), you cannot recover a density on $Y$; instead, $Y$ lives on a lower-dimensional manifold.

Change of variables shows up throughout probabilistic ML. Deriving the distribution of a sum, ratio, or transformation of known variables (e.g., $Y = e^X$ for $X$ Gaussian gives a log-normal). Converting between scale parameters (when modeling $\sigma$ vs $\log \sigma$ or $\tau = 1/\sigma^2$). Any time we substitute a parameterization, the Jacobian keeps the probabilities honest.

The marquee application is normalizing flows: stack a sequence of invertible transformations $f_1, f_2, \dots, f_K$ to map a simple base distribution (standard Gaussian) to a complex target. Each layer contributes a log-det-Jacobian term, and the log-likelihood of the target becomes $\log p_X(\mathbf{x}) = \log p_Z(f^{-1}(\mathbf{x})) + \sum_k \log|\det J_{f_k^{-1}}|$. Flows are the clearest modern use of change of variables — invertibility and tractable Jacobians are the whole design constraint.

Worked example — log-normal from Gaussian: If $X \sim \mathcal{N}(0,1)$ and $Y = e^X$, then $X = \ln Y$ with $dx/dy = 1/y$, so $p_Y(y) = \frac{1}{y\sqrt{2\pi}}\exp(-(\ln y)^2/2)$ for $y > 0$. The Jacobian factor $1/y$ is essential — dropping it would give a non-normalizable density. This is why normalizing flows track every $\log|{\det J}|$ term.