5 · TF-IDF item profiles
When items are described by text or tags, not every word is equally informative. TF-IDF down-weights tags everything shares and up-weights the rare, distinctive ones — then cosine similarity finds genuinely similar items.
TF-IDF turns each item's text/tags into a weighted vector: a tag's weight rises with how often it appears in that item (term frequency) and falls with how many items contain it (inverse document frequency). Generic tags shared by everything get near-zero weight; rare, distinctive tags dominate. Cosine similarity over these vectors finds truly similar items.
Without this:
Use raw word counts and a generic tag like 'movie' (present in every description) counts as much as a rare, telling tag like 'heist'. Your similarities collapse toward 'everything is similar' because the common words drown out the signal.
Genre one-hots are clean but coarse — real items come with free text: a synopsis, a tag cloud, a list of keywords. To turn that into a feature vector you need a way to weight each word, because not all words carry equal signal. The word 'movie' appears in every film's description and tells you nothing; the word 'heist' appears in a handful and is highly discriminative.
TF-IDF (Term Frequency × Inverse Document Frequency) is the classic weighting that captures exactly this intuition:
- TF — how often a term appears in this item. More mentions → more relevant to this item.
- IDF —
log(N / df), whereNis the number of items anddfis how many items contain the term. A term in every item hasdf = N, soIDF ≈ 0and its weight is crushed; a term in one item has a large IDF and keeps its weight.
The product gives each (item, term) a weight that is high only when the term is frequent here but rare overall — the hallmark of a distinctive feature. Stack these into a matrix (rows = items, columns = vocabulary) and you have rich content vectors. Then, exactly as in the previous lesson, cosine similarity between two item vectors measures how much their distinctive vocabulary overlaps — and ranks the most similar items for any query.
scikit-learn's TfidfVectorizer does the tokenizing, counting, and IDF weighting in one call, and cosine_similarity gives the full item-to-item similarity matrix. Below we describe six films by short tag strings — deliberately putting the generic word 'movie' in every one — and watch TF-IDF strip its weight while the sci-fi neighbors of a query film rise to the top.
Python (in browser)
TfidfVectorizer + cosine_similarity: the generic tag 'movie' (in every item) gets a tiny weight, so the query's true sci-fi neighbors rank top.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Read-along: how a real system scales this — stop-words, min_df, n-grams, and a pre-fit cosine NearestNeighbors index for fast 'more like this'.
A tag appears in EVERY item's description. What does TF-IDF do to its weight, and why?
- TF-IDF weights a term high only when it is frequent in this item (TF) but rare across the catalog (IDF = log(N/df)).
- Generic tags shared by every item get near-zero weight, so they don't dominate similarity — distinctive tags do.
- sklearn's TfidfVectorizer + cosine_similarity turns short text/tags into item vectors and finds the nearest items for 'more like this'.
'Because you read this article', tag-based 'related products', and synopsis-driven 'more like this' rows are all TF-IDF (or its embedding successors) over item text.
If you remove it: Weight all words equally (raw counts) and common boilerplate dominates similarity, so every item looks similar to every other — the recommendations become useless.