3 · Multiple linear regression
Multiple features turn one slope into a vector — and each coefficient means 'effect of this feature, holding all others constant'.
Multiple features turn ŷ = wx + b into ŷ = w₁x₁ + w₂x₂ + ... + wₙxₙ + b — vector dot product.
Without this:
Real datasets always have many features; you can't model house price from square footage alone.
The jump from simple (one feature) to multiple linear regression (many features) is mostly notational — the model is still linear, just in a higher-dimensional space.
In matrix form: ŷ = Xw + b (or ŷ = Xw if you absorb the bias b as an extra column of ones in X).
The closed-form solution extends to the Normal Equation:
w = (XᵀX)⁻¹ Xᵀy
This solves for the weight vector w ∈ ℝᵖ that minimises the sum of squared residuals across all p features simultaneously. Two practical notes:
- Use
np.linalg.solve(X.T @ X, X.T @ y)instead ofnp.linalg.inv(X.T @ X) @ X.T @ y—solveis numerically more stable (avoids explicit inversion). - If two features are near-perfectly correlated (multicollinearity), XᵀX is nearly singular — no unique solution exists. Ridge regression (Lesson 5) is the standard fix.
Interpreting coefficients is the real power of linear models: wᵢ is the expected change in ŷ when feature xᵢ increases by one unit, holding all other features constant. This "partial derivative" interpretation is why econometricians say "controlling for X" and why you must scale features before comparing coefficient magnitudes.
Python (in browser)
Each coefficient estimates 'effect per unit increase in that feature, all else equal'. The recovered values should be close to the true generating parameters.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
`np.linalg.solve(A, b)` finds w such that Aw = b without explicitly inverting A — faster and numerically more stable than `np.linalg.inv(A) @ b`.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
Unscaled: sqft coefficient is tiny (0.15) because sqft spans 500–3000. Scaled: the coeff is large because 1 std of sqft is ~700 sqft. StandardScaler makes magnitudes comparable.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
The 'holding all others constant' interpretation is what makes linear models interpretable — and what logistic regression / GLMs inherit from OLS.
If two features have correlation ρ = 0.99, what is the risk for a linear regression's coefficients?
- Multiple regression is ŷ = Xw + b — same OLS principle, now w is a vector. Closed form: w = (XᵀX)⁻¹Xᵀy, solved with `np.linalg.solve`.
- Each coefficient wᵢ = ∂ŷ/∂xᵢ — the effect of feature i, holding all others constant. This is the 'controlling for X' of regression analysis.
- Apply `StandardScaler` before comparing coefficient magnitudes — raw coefficients reflect units, not importance.
- Highly correlated features (ρ > 0.9) cause multicollinearity — coefficients blow up. Diagnose with VIF; fix with ridge regression or feature removal.
Every tabular regression problem starts with multiple linear regression. Time series often baseline with ARX models which are multiple linear regression with lagged features.
If you remove it: Without it you'd model only single-variable relationships — far too limited for any real problem.