17 · Metrics & baselines
Before you celebrate any model, measure its error with MAE / RMSE / MAPE and compare it against the two baselines every forecast must beat: naive and seasonal-naive.
A forecast is only good relative to a baseline. MAE / RMSE / MAPE quantify the error, and the naive (last value) and seasonal-naive (same point one season ago) baselines set the bar — if your fancy model can't beat them, it adds nothing.
Without this:
Report an RMSE with no baseline and you have no idea whether the number is good; teams routinely ship models that lose to a one-line seasonal-naive forecast.
Three error metrics dominate forecasting, and each tells a slightly different story:
- MAE (Mean Absolute Error) —
mean(|y - ŷ|). In the same units as the data, easy to explain ("off by 4 units on average"), and robust to outliers. - RMSE (Root Mean Squared Error) —
sqrt(mean((y - ŷ)²)). Also in data units but squares the errors first, so it punishes large misses harder than MAE. Use it when big errors are especially costly. - MAPE (Mean Absolute Percentage Error) —
mean(|y - ŷ| / |y|) × 100. Scale-free (a percentage), great for comparing across series of different magnitudes — but it explodes when actuals are near zero (division by a tiny number) and is undefined at exactly zero. Never trust MAPE on intermittent or zero-heavy data.
A single number means nothing on its own. You must compare it against baselines — trivial forecasts a real model has to beat:
- Naive: tomorrow = today. The forecast for
t+his just the last observed value. Shockingly hard to beat on random-walk-like series. - Seasonal-naive:
t+hequals the value one full season ago (e.g. "next Monday = last Monday" for period 7). On seasonal data this is the bar to clear.
The code below implements all three metrics from scratch, builds the two baselines on a hold-out, and prints each baseline's error — the literal number your model in production must come in under.
Python (in browser)
MAE / RMSE / MAPE implemented from scratch, plus the naive and seasonal-naive baselines on a hold-out — the error bar your model must beat.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
You're forecasting daily product returns, a series that is often exactly 0 on quiet days. Which metric is the WORST choice and why?
- MAE is robust and in data units; RMSE (also in units) punishes large errors harder; MAPE is scale-free but breaks near zero.
- A metric is meaningless without a baseline: naive (last value) and seasonal-naive (value one season ago) set the bar to beat.
- On seasonal data seasonal-naive is the bar; if your model can't beat it, the model adds no value.
Every forecasting competition, demand-planning review, and monitoring dashboard reports model error against naive/seasonal-naive baselines.
If you remove it: Without a baseline you can't tell a good RMSE from a bad one, and you risk shipping a model that loses to a one-line forecast.