Systems of Linear Equations

A system of linear equations is a collection of equations in which each unknown appears only to the first power and is never multiplied by another unknown. A familiar example is $2x + 3y = 7$ and $x - y = 1$. The word *linear* means each equation traces out a flat object: a line in two dimensions, a plane in three, and a Hyperplaneⓘ in higher dimensions.

Why do machine learners care? Because fitting a linear model is literally solving a system. Given training examples $(\mathbf{x}_i, y_i)$, the least-squares regression problem asks for a weight Vectorⓘ $\mathbf{w}$ such that $X\mathbf{w} \approx \mathbf{y}$, where $X$ is a Matrixⓘ stacking the feature vectors as rows. When $X$ is square and invertible the system has a unique solution; when it is not (the common case in ML), we fall back on approximation via the *normal equations* — still a linear system.

Every linear system has one of three possible outcomes: a unique solution, no solution, or infinitely many solutions. Geometrically in 2D, two lines can meet at a single point (unique), be parallel and never meet (none), or coincide entirely (infinite). This trichotomy generalizes perfectly to higher dimensions and is the reason we care about concepts like rank and consistency later on.

A compact way to write a linear system is $A\mathbf{x} = \mathbf{b}$, where $A$ is the coefficient matrix, $\mathbf{x}$ is the vector of unknowns, and $\mathbf{b}$ is the constants vector. This notation is not just shorthand — it reveals structure. Many properties of the solution set (existence, uniqueness) can be read off from $A$ alone, before we ever start solving. That is the power of moving from equations to matrices.

Every line in 2D can be written in slope-intercept form $y = mx + b$. Drag the sliders below to see how changing the slope $m$ and the intercept $b$ rotates and shifts the line. When you have *two* such lines, their linear system asks where they cross.

Worked example — a unique solution: Solve $2x + 3y = 7$ and $x - y = 1$. From the second equation, $x = y + 1$. Substituting and isolating $y$:

Now do it yourself. Each input below should be algebraically equivalent to the line above it — type your simplification and press Enter. The system checks equivalence symbolically (and numerically when symbolic simplification is ambiguous), so any valid intermediate form is accepted.

So $x = y + 1 = 2$. The unique intersection point is $(2, 1)$ — check: $2\cdot 2 + 3\cdot 1 = 7$ ✓ and $2 - 1 = 1$ ✓.

Worked example — no solution (parallel lines): Consider $x + y = 2$ and $2x + 2y = 5$. Divide the second by 2:

But the first equation says $x + y = 2$. These cannot both hold, so the system is inconsistent — geometrically two parallel lines that never meet.

ML connection — normal equations: For the regression system $X\mathbf{w} = \mathbf{y}$ with $X \in \mathbb{R}^{100 \times 5}$, there are 100 equations but only 5 unknowns — usually no exact solution exists. The normal equations $X^\top X \mathbf{w} = X^\top \mathbf{y}$ are a *square* $5\times 5$ linear system whose unique solution (when $X$ has full column rank) is the least-squares fit.

Solving a system by hand usually means Gaussian elimination: repeated row operations that simplify $A$ into an upper-triangular form, followed by back-substitution. We will formalize this in the next lesson. The key takeaway for now: a linear system is a geometric intersection problem, a matrix equation, and a solvable algorithmic puzzle — three viewpoints on the same object.