Vector Spaces

So far we have worked with concrete vectors in $\mathbb{R}^n$. The true power of linear algebra, however, comes from abstracting: a vector space over the real numbers is any set $V$ equipped with an addition and a scalar multiplication that satisfy eight familiar axioms (associativity, commutativity of addition, existence of a zero vector, existence of additive inverses, compatibility of scalar multiplication, two distributive laws, and the identity $1 \cdot \mathbf{v} = \mathbf{v}$).

This abstraction is not academic hairsplitting. Once a set obeys the axioms, *every theorem of linear algebra applies automatically*. The space of polynomials of degree ≤ $n$, the space of $m \times n$ matrices, the space of continuous functions on $[0, 1]$ — all are vector spaces, and concepts like linear combinations, independence, and bases transfer wholesale.

A subspace $U \subseteq V$ is a subset that is itself a vector space under the inherited operations. Concretely, $U$ must contain the zero vector and be closed under addition and scalar multiplication: for any $\mathbf{u}_1, \mathbf{u}_2 \in U$ and any scalar $\lambda$, both $\mathbf{u}_1 + \mathbf{u}_2$ and $\lambda \mathbf{u}_1$ must still lie in $U$. A line through the origin in $\mathbb{R}^2$ is a subspace; a line that misses the origin is not.

Two subspaces attached to every matrix $A$ are cornerstones of ML. The column space (or *range*) is the span of $A$'s columns — exactly the set of $\mathbf{b}$ for which $A\mathbf{x} = \mathbf{b}$ has a solution. The null space (or *kernel*) is the set of $\mathbf{x}$ with $A\mathbf{x} = \mathbf{0}$ — the directions in input space that $A$ flattens to zero. The fundamental theorem of linear algebra relates their dimensions: the rank-nullity theorem states that for $A \in \mathbb{R}^{m\times n}$, $\dim(\text{col}(A)) + \dim(\text{null}(A)) = n$.

Worked example — NOT a subspace: The set $W = \{(x, y) : x \geq 0\}$ is the right half-plane. It contains $(0,0)$ ✓, and sums of non-negative-$x$ vectors stay non-negative ✓, BUT $\lambda = -1$ times $(1, 0)$ gives $(-1, 0) \notin W$. Not closed under scalar multiplication, so not a subspace — it's only closed under non-negative scalars.

Why should a machine learner care about subspaces? Because features live in them. When PCA reduces a dataset to its top $k$ directions, it projects onto a $k$-dimensional subspace. When a neural network's hidden layer has rank-deficient weights, certain input directions are collapsed and forever invisible to the rest of the network. Understanding subspaces is the key to diagnosing representational capacity.