1 · Simple linear regression
Fit a line to two variables — the foundation every other supervised model is compared against.
Fit a line ŷ = wx + b that minimises the sum of squared errors — your first ML model in two lines of code.
Without this:
Without linear regression you have no baseline; every more complex model must beat it before it's worth deploying.
Linear regression is the simplest supervised learning model — and arguably the most important. Given pairs (x, y), it finds the straight line ŷ = wx + b that passes "as close as possible" to the data points. "As close as possible" means minimising the Mean Squared Error (MSE):
MSE = (1/n) · Σ(yᵢ − ŷᵢ)²
The parameter w is the slope (how much ŷ changes per unit increase in x) and b is the intercept (the value of ŷ when x = 0). Together they define every possible straight line in 2D — so fitting the model is a search over all possible (w, b) pairs for the one that makes predictions closest to the labels.
Why MSE? Because squared errors penalise large deviations more than small ones, and the squaring makes the loss differentiable everywhere — which makes optimisation tractable. The unique minimum of MSE has a closed-form solution called Ordinary Least Squares (OLS), which we'll derive by hand in this lesson.
The four OLS assumptions we carry forward from statistics:
- Linearity — the true relationship between x and y is linear.
- Independence — residuals are not correlated with each other.
- Homoscedasticity — the variance of the residuals is constant across all x.
- Normality — residuals are approximately normally distributed (needed for valid inference / confidence intervals, less critical for pure prediction).
Python (in browser)
`model.coef_` is a 1-D array even for a single feature — index it as `[0]`. The recovered w ≈ 2.5 and b ≈ 1.0 confirm the model found the true generating parameters.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
`model.score(X, y)` returns R² — the proportion of variance explained. On a clean linear dataset both R² values are close to 1.0.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
The OLS formula w = cov(x,y)/var(x) comes from setting ∂(MSE)/∂w = 0 and solving. Both approaches give identical results — sklearn uses the same derivation internally.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
The OLS assumptions determine whether your coefficient standard errors and p-values are trustworthy — not whether the line itself is the best linear fit.
If sklearn returns `coef_ = [3.2]` and `intercept_ = -1.5`, what does the model predict for x = 4?
Linear Regression: Problem Formulation — the MML chapter that derives ŷ = wx + b and the MSE loss from first principles.
- The model is ŷ = wx + b; the loss is MSE = (1/n)·Σ(yᵢ − ŷᵢ)². Minimising MSE gives the OLS solution: w = cov(x,y)/var(x), b = ȳ − w·x̄.
- sklearn needs X as shape (n, 1) for a single feature — always `X.reshape(-1, 1)` before fitting.
- R² measures the proportion of variance explained: 0 = no better than the mean, 1 = perfect. Always evaluate on a held-out test set.
- Linear regression is the universal baseline — any more complex model must beat it to justify its extra cost.
Every supervised regression task starts here. Logistic regression is linear regression in feature space, then a sigmoid. Neural network linear layers ARE linear regression. Even gradient boosting fits linear regressions in each weak learner's leaves.
If you remove it: You'd have no reference point — 'this model achieves 92% R²' means nothing without 'the linear baseline achieves 87%'.