2 · Population vs sample
The window you look through and the world it shows — size and method determine what you can claim.
A sample is a window into a population — the size and method of that window determine what you can claim.
Without this:
Without distinguishing the two, you'll confuse 'the training data did X' with 'the world does X' — and ship models that fail on production traffic.
Two terms that sound simple but cause enormous confusion in ML practice:
- Population: the complete set of all units (people, transactions, documents, images…) you're ultimately interested in. You almost never observe the full population.
- Sample: the subset you actually measure. Your training data is a sample. Your test set is a sample.
Once you have a sample, you compute statistics (x̄, s) — these are random variables that vary from sample to sample. The corresponding population quantities (μ, σ) are called parameters — they are fixed but unknown numbers you're trying to estimate.
The whole machinery of inferential statistics exists because you only ever see samples.
Python (in browser)
Each sample gives a different x̄. This variation IS the sampling distribution — the central object of inferential statistics.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Not all samples are equally trustworthy. The method of sampling is critical:
| Method | Description | Bias risk | |---|---|---| | Simple random | Every unit equally likely | Low | | Stratified | Random within defined subgroups | Low (better coverage) | | Cluster | Random groups, measure all in group | Medium | | Systematic | Every k-th unit | Low–medium | | Convenience | Whatever is easy to collect | High |
ML datasets are almost always convenience samples — whatever the company happened to collect. This is the root cause of distribution shift.
Python (in browser)
Convenience sampling can produce a sample mean that is dramatically far from the population mean — no amount of model sophistication fixes a biased dataset.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
If μ is the population mean and x̄ is a sample mean, which one is a random variable?
- **Population** = every unit of interest; **sample** = the subset you measure. Parameters (μ, σ) belong to the population; statistics (x̄, s) belong to the sample.
- Sample statistics are random variables — they vary across samples. This variation is the foundation of every confidence interval and hypothesis test.
- Sampling method determines bias. Convenience samples (most ML data) are systematically skewed away from the true distribution.
Your training set is a SAMPLE from the production distribution; your test accuracy is a STATISTIC estimating the true generalization performance (a parameter). Domain shift is just 'sample doesn't match population'.
If you remove it: You can't reason about why your 95%-train-accuracy model gets 71% in production without the population/sample frame.