Basis and Rank

A basis of a vector space $V$ is a set of vectors that is both linearly independent and spanning — every vector in $V$ can be written as a unique linear combination of basis elements. Every basis of a given finite-dimensional space has the same number of vectors, and that number is the dimension of the space. This is remarkable: dimension does not depend on which basis you chose, only on the space itself.

The most familiar basis of $\mathbb{R}^n$ is the standard basis $\{\mathbf{e}_1, \dots, \mathbf{e}_n\}$, where $\mathbf{e}_i$ has a 1 in position $i$ and zeros elsewhere. But infinitely many other bases exist. For instance, $\{(1,1), (1,-1)\}$ is a perfectly valid basis of $\mathbb{R}^2$. Choosing a *different* basis can dramatically simplify a problem — this is the motivation for eigendecomposition and PCA.

The coordinates of a vector $\mathbf{v}$ in a basis $B = \{\mathbf{b}_1, \dots, \mathbf{b}_n\}$ are the unique scalars $c_1, \dots, c_n$ satisfying $\mathbf{v} = \sum c_i \mathbf{b}_i$. The same geometric vector has different coordinate representations in different bases — a fact that is easy to forget but crucial for transformations like change-of-basis matrices.

The rank of a matrix $A$ is the dimension of its column space — equivalently, the number of linearly independent columns, the number of pivots in its row-echelon form, or (a beautiful identity) the dimension of its row space. A key fact: $\text{rank}(A) = \text{rank}(A^\top)$, i.e. row rank equals column rank. This identity has real consequences: for a tall matrix $A \in \mathbb{R}^{m \times n}$ with $m > n$, the rank is at most $n$, because there are only $n$ columns to be independent.

Worked example — computing rank by elimination: For $A = \begin{pmatrix}1 & 2 & 3\\2 & 4 & 7\\3 & 6 & 10\end{pmatrix}$: R2 ← R2 − 2R1 gives $(0, 0, 1)$; R3 ← R3 − 3R1 gives $(0, 0, 1)$; R3 ← R3 − R2 gives $(0, 0, 0)$. Two non-zero rows remain → rank = 2. The third column is *not* in the span of the first two's linear dependency pattern, so the rank is 2 rather than 1.

Worked example — coordinates in a non-standard basis: In the basis $B = \{(1,1), (1,-1)\}$, find the coordinates of $\mathbf{v} = (4, 2)$. Solve $c_1(1,1) + c_2(1,-1) = (4,2)$: $c_1 + c_2 = 4$, $c_1 - c_2 = 2$. Adding: $2c_1 = 6 \Rightarrow c_1 = 3$, then $c_2 = 1$. So $[\mathbf{v}]_B = (3, 1)$ — the same geometric vector has different coordinates in different bases.

Rank is the central diagnostic in many ML pipelines. A full-rank design matrix lets ordinary least-squares produce a unique solution. A low-rank weight matrix in a trained neural network is often a sign that the model has compressed information into fewer effective dimensions — the basis of modern techniques like LoRA (Low-Rank Adaptation), which adapts large models by adding a small, low-rank update rather than retraining all parameters.