Linear Independence

A set of vectors $\{\mathbf{v}_1, \mathbf{v}_2, \dots, \mathbf{v}_k\}$ is linearly independent if the only way to form the zero vector as $\lambda_1 \mathbf{v}_1 + \lambda_2 \mathbf{v}_2 + \dots + \lambda_k \mathbf{v}_k = \mathbf{0}$ is with all $\lambda_i = 0$. If any non-trivial combination gives zero, the vectors are linearly dependent and (at least) one is a redundant combination of the others.

Independence is the mathematical statement of *non-redundancy*. Intuitively, $k$ independent vectors point in *genuinely different* directions, each contributing a dimension that the others cannot fake. Dependent vectors, by contrast, have overlap: one of them can be expressed using the rest, so removing it loses no information.

There is a clean algorithmic test: form the matrix $A$ whose columns are the vectors $\mathbf{v}_i$. The vectors are linearly independent iff the homogeneous system $A\mathbf{x} = \mathbf{0}$ has only the trivial solution $\mathbf{x} = \mathbf{0}$ — equivalently, iff $A$ has full column rank, or (in the square case) iff $\det(A) \neq 0$. You can always run Gaussian elimination and count pivots: *number of pivots = number of independent columns*.

Some quick facts that often show up in problems. Any set containing the zero vector is automatically dependent. Any set with more vectors than the dimension of the surrounding space is automatically dependent: you cannot have five independent vectors in $\mathbb{R}^4$. A single non-zero vector is always independent.

Worked example — dependent pair: Check if $\mathbf{v}_1 = (1, 2)$ and $\mathbf{v}_2 = (2, 4)$ are independent. Solve $\lambda_1(1,2) + \lambda_2(2,4) = (0,0)$. From the first coordinate, $\lambda_1 + 2\lambda_2 = 0$, so $\lambda_1 = -2\lambda_2$. Taking $\lambda_2 = 1, \lambda_1 = -2$ gives a non-trivial combination summing to zero — dependent. Indeed, $\mathbf{v}_2 = 2\mathbf{v}_1$.

In ML, linear independence of features matters enormously. If two columns of your design matrix $X$ are linearly dependent (e.g., temperature_celsius and temperature_fahrenheit), then $X^\top X$ is singular and the ordinary least-squares solution is not unique — the model weights are ill-defined. This is the issue behind *multicollinearity* in regression, and it is also why regularization (ridge, lasso) is so often necessary in practice.