14 · Beyond accuracy: diversity & coverage
A perfectly accurate recommender can still be a bad one — recommending five near-identical items, ignoring the catalog, trapping users in a filter bubble. Measure coverage, diversity, and novelty too.
Accuracy is necessary but not sufficient. A recommender that nails NDCG by serving five near-duplicate blockbusters is boring and useless — measure catalog COVERAGE (how much of the catalog ever gets recommended), intra-list DIVERSITY (1 − average pairwise item similarity), and NOVELTY (self-information of how unpopular the items are), and re-rank to trade a little accuracy for them.
Without this:
Optimize accuracy alone and you build a filter bubble: every user sees the same handful of popular, interchangeable items, the long tail never surfaces, and engagement quietly decays.
A recommender can score beautifully on NDCG and still be a bad product. Three failure modes that accuracy metrics are blind to:
Filter bubble / over-specialization. If you only show items extremely similar to what a user already liked, every recommendation is a near-duplicate. Accurate — and stultifying. Discovery dies.
Popularity bias / poor coverage. Models trained on engagement learn to recommend the same few blockbusters to everyone, because popular items have the most signal. The catalog's long tail is never shown, hurting creators, inventory, and the very serendipity that makes a feed feel alive.
So we measure quality beyond accuracy:
- Catalog coverage — the fraction of distinct items that appear in anyone's recommendations. Low coverage = a system that recycles the same shelf.
- Intra-list diversity — for a single user's list,
1 − (average pairwise similarity)between its items. High = a varied list; low = five flavors of the same thing. - Novelty — reward recommending the less popular. Using self-information
-log2(popularity), a rarely-seen item carries more 'surprise' than a chart-topper; averaging it over a list scores how off-the-beaten-path the recommendations are.
These trade off against accuracy. The standard tool is re-ranking: start from the accuracy-ranked list, then greedily build the final list by balancing each candidate's relevance against how dissimilar it is to what you've already picked (the idea behind Maximal Marginal Relevance). You sacrifice a sliver of predicted relevance to break the bubble.
Below we take recommendation lists plus an item-item similarity matrix, compute coverage and intra-list diversity, then run a small MMR-style re-rank that trades a little accuracy for a markedly more diverse list.
Python (in browser)
Catalog coverage + intra-list diversity, then an MMR re-rank that trades a sliver of predicted relevance for a noticeably more diverse list.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Novelty via self-information (-log2 popularity) rewards surfacing the long tail; frameworks like RecBole report coverage, popularity, and entropy beside NDCG.
A recommender posts an excellent NDCG@10 but every user's list is five near-identical items from the same popular sub-genre. What is the problem and a standard fix?
- Accuracy is necessary but not sufficient — a high-NDCG list of near-duplicates is a filter bubble.
- Measure catalog coverage, intra-list diversity (1 − mean pairwise similarity), and novelty (-log2 popularity) alongside accuracy.
- Re-ranking (MMR) trades a sliver of predicted relevance for diversity/coverage — aim for the Pareto frontier, not pure accuracy.
YouTube, Spotify, and news feeds explicitly tune for diversity/novelty to fight filter bubbles; re-ranking layers (MMR, DPP) sit on top of the accuracy model in production stacks.
If you remove it: Optimizing accuracy alone builds an echo chamber: the long tail vanishes, creators starve, and users churn from sameness despite 'good' offline metrics.