Time-series feature regression
statsmodelsProphetForecasting
Lag features, rolling means, seasonal decomposition, and Fourier seasonality components all feed into linear regression to forecast retail demand, energy load, traffic.
Ch. 9 · Linear Regression
Linear Regression: Problem Formulation
Linear Regression
Ch. 9
y^=w⊤x+b
the linear model
Linear regression is the first and most important supervised learning algorithm. The model asserts that the target y is a linear function of features mathbf{x} in mathbb{R}^D plus Gaussian noise: y = mathbf{x}^top boldsymbolbeta + varepsilon, quad varepsilon sim mathcal{N}(0, sigma^2). Given trainin
⏱ 2min read◇ 2vizs.✓ 5exercises
Before you start
Foundations you'll lean on here
A quick refresher on what this lesson assumes. Tap any card for a plain-language explanation and a tiny worked example.
★Big idea
Linear regression: ; closed-form .
Without this:
Without the closed-form and MLE view of least squares, you cannot derive Ridge, Lasso, Bayesian regression, Kalman filters, or the linear output layer of a neural network in any principled way.
Linear regression is the first and most important supervised learning algorithm. The model asserts that the target is a linear function of features plus Gaussian noise: Given training data , we stack features into the design matrix and targets into , giving the matrix form .
The least squares solution minimizes the sum of squared residuals: Setting the gradient to zero gives the normal equations . When is invertible, The term is the Moore-Penrose pseudo-inverse — the closest thing to an inverse when is tall and thin.
From a probabilistic view, this same is the MLE: under Gaussian noise with fixed , maximizing is equivalent to minimizing squared error. Linear regression is a rare case where ERM and MLE coincide exactly, and both admit a closed-form solution — no iterative optimization needed.
Real-data companion
Fit a line through 20 real housing samples
Housing (20-row sample)
Rooms vs income — the emerald line is the closed-form least-squares solution .
price tier lowprice tier high
X axis
Y axis
mean X5.75
mean Y3.17
var X3.46
var Y1.46
cov(X,Y)2.20
λ₁4.88
λ₂0.04
Least squares fit
The regression line minimizes the sum of squared vertical distances to the points — the geometry of MLE for Gaussian noise.
↻
Feature engineering extends the reach dramatically. Replacing with gives polynomial regression — still linear *in the parameters*, hence 'linear regression'. Radial basis functions, splines, Fourier features, and even random neural-network features all fit this template: choose a feature map , then solve a linear least-squares problem. This is the generalized linear model view and the foundation of kernel methods.
When (more features than samples) or when features are correlated, is singular and the solution is not unique. Ridge regression adds an penalty: — always invertible, numerically stable, and equivalent to MAP with a Gaussian prior. Lasso ( penalty) yields sparse solutions and is the workhorse for feature selection. Both are convex and are implemented in every ML library on Earth.
Worked example — normal equations on 3 points: Fit to . The design matrix is , so and . Solving gives , . Prediction at : .
1/2
Normal equations vs sklearn
Python (in browser)
Expected: Both vectors agree (modulo numerical noise)
main.pypython
Loading editor…
Ready — click Run to execute in your browser.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Squared loss vs residual
Each residual contributes its square to the loss; large errors dominate, which makes least squares sensitive to outliers.
Worked example — ridge shrinkage: On the same problem, set so we solve , i.e. . Cramer's rule gives , , . The slope shrank from to — bias for variance, exactly as advertised.
1/2
1/2
1/3
In the wild
Where this shows up in production
Each card is a real system, model, or technique that rests on this lesson.
Recommendation system feature scoring
RecsysProduction
Linear models over hand-engineered features (collaborative + content + popularity) are still the production fallback when deep models are too slow or unexplainable.
ElasticNet for genomics / NLP word counts
sklearnGenomicsNLP
When (e.g. millions of SNPs, tens of thousands of TF-IDF columns), L1+L2 hybrid regularization shrinks irrelevant features and stabilizes the rest.
python
from sklearn.linear_model import ElasticNetCV
model = ElasticNetCV(l1_ratio=[.1,.5,.9], cv=5).fit(X, y)PyTorch nn.Linear
PyTorch
The first and last layer of nearly every neural network is nn.Linear — a linear regression. Everything in between learns the right .
python
import torch.nn as nn
fc = nn.Linear(in_features=512, out_features=1)Exercises
Concept Challenge
Put your understanding to the test. Score + streak + speed all count.
Linear Regression: Problem Formulation
5 rounds1000 max pts
Active recall
Recap flashcards
1 / 3
Concept
Linear regression
Tap the card to reveal
Confirm you've got it
3 quick questions. Get 2 right to mark this lesson complete.