3 · Cold-start & sparsity
The two problems that define recommender engineering: a near-empty matrix makes similarities unreliable, and brand-new users/items have no data at all.
Two structural problems shape every recommender: sparsity (with <1% of entries observed, two users rarely overlap, so similarities are noisy) and cold-start (a brand-new user or item has zero interactions, so pure collaborative filtering can't say anything).
Without this:
Ignore these and your recommender works great in a demo and falls apart in production — silent on new items, unreliable for casual users, and dominated by a few popular blockbusters.
Two problems are so fundamental that the rest of the field is largely a response to them.
Sparsity. Real matrices have far less than 1% of entries filled. The consequence is brutal for collaborative filtering: to judge whether two users are 'similar', you compare the items they've both rated — but with sparse data, two users may share only one or two items in common (or zero), making any similarity estimate noisy or undefined. Popular items get rated a lot and dominate; the long tail is starved of signal. Much of recsys (matrix factorization especially) exists to extract reliable structure from this near-emptiness.
Cold-start. Collaborative filtering learns only from past interactions — so it's helpless when there are none:
- New user: someone who just signed up has rated nothing. Who are they similar to? Nobody yet.
- New item: a movie added today has no ratings. Whose taste does it match? Unknown.
- New system: a brand-new product has an empty matrix entirely.
The escapes are exactly the other families: content-based methods (Chapter 2) sidestep item cold-start by using item features (a new movie still has a genre and cast), and popularity / onboarding handles new users until they generate signal. Production systems are hybrids precisely because no single method survives both problems.
Below we make sparsity concrete: as we hide more of a matrix, the number of users who share enough items to be comparable collapses — a direct measurement of why naive collaborative filtering struggles.
Python (in browser)
As the matrix gets sparser, the share of user pairs with enough overlap to be comparable collapses — the quantitative root of the cold-start/sparsity problem.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
A brand-new movie is added to the catalog with zero ratings. Why can't pure collaborative filtering recommend it, and what fixes this?
- Sparsity (>99% missing) makes user/item overlaps tiny, so memory-based similarities are noisy or undefined.
- Cold-start: new users/items/systems have no interactions, so pure collaborative filtering is silent.
- Content-based features fix item cold-start; popularity/onboarding fix new users; production = hybrids.
Every streaming/e-commerce platform faces cold-start daily (new releases, new signups) and engineers explicit onboarding + content fallbacks for it.
If you remove it: Ignoring cold-start/sparsity yields a recommender that only works for power users on popular items — useless for growth.