Matrices

A matrix is a rectangular array of numbers arranged into rows and columns. We denote an $m \times n$ matrix as $A \in \mathbb{R}^{m \times n}$, where $m$ is the number of rows and $n$ is the number of columns. The entry in row $i$ and column $j$ is written $a_{ij}$. In machine learning, a dataset with $m$ examples and $n$ features is usually stored as such a matrix — often called the design matrix.

Matrices support three fundamental operations. Addition and scalar multiplication work entry-wise and require matching dimensions (for addition). Matrix multiplication, in contrast, is subtler. The product $AB$ is defined only when the number of columns of $A$ equals the number of rows of $B$, and the $(i,j)$ entry of $AB$ is the dot product of row $i$ of $A$ with column $j$ of $B$: $(AB)_{ij} = \sum_{k} a_{ik} b_{kj}$.

Matrix multiplication is not commutative: in general $AB \neq BA$, even when both products are defined. It is, however, associative ($(AB)C = A(BC)$) and distributive over addition. These algebraic laws are what let us factor, rearrange, and decompose expressions in ML — for example, when deriving the normal equations $(X^\top X)\mathbf{w} = X^\top \mathbf{y}$.

Special matrices show up everywhere. The identity matrix $I_n$ has ones on the diagonal and zeros elsewhere; it satisfies $IA = AI = A$. The transpose $A^\top$ flips rows and columns, and $(AB)^\top = B^\top A^\top$. A matrix is symmetric if $A = A^\top$ (covariance matrices are always symmetric), diagonal if only the diagonal is non-zero, and orthogonal if $A^\top A = I$ (orthogonal matrices preserve lengths and angles — think rotations and reflections).

The inverse $A^{-1}$, when it exists, satisfies $A A^{-1} = A^{-1} A = I$. A matrix is invertible (also called *non-singular*) iff it is square and its columns are linearly independent. Not every matrix has an inverse — for the ones that do not, we reach for generalized inverses like the Moore–Penrose pseudo-inverse, which is the beating heart of least-squares solutions.