10 · TF-IDF: intuition + math
Understand why raw word counts mislead — and how TF-IDF weights rare, document-specific words high while silencing common noise tokens.
TF-IDF weights raw word counts by inverse document frequency — rare, document-specific words score high; common 'noise' words score low.
Without this:
BOW treats 'the' the same as 'antibiotic' — your vector is dominated by uninformative tokens.
Bag of Words counts how many times each word appears in a document. That sounds reasonable — until you realise that extremely common words like "the", "is", and "a" accumulate huge counts in every document without telling you anything useful about what the document is about.
BOW double-counts common words. A document with 500 occurrences of "the" doesn't contain more information about "the" than a document with 50. Those raw counts just add noise.
TF-IDF (Term Frequency–Inverse Document Frequency) fixes this by multiplying two factors:
- TF(t, d) — how often term t appears in document d (or a normalized version of it).
- IDF(t) — how much information the term carries across the whole corpus. Words appearing in almost every document carry almost no discriminative signal — their IDF is close to zero.
The result: a word that appears many times in one document but rarely in others gets a high TF-IDF score. A word that appears everywhere gets a score near zero — automatically suppressed, no stopword list required.
The formula:
TF(t, d) = count(t, d) / total_tokens(d) # normalized term frequency
IDF(t) = log( N / DF(t) ) # N = total docs, DF = docs containing t
TF-IDF = TF(t, d) * IDF(t)
sklearn's TfidfVectorizer uses a smoothed variant to avoid division by zero (+1 inside the log and in the denominator) and L2-normalises each document row so document length doesn't skew comparisons.
Python (in browser)
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Four reasons TF-IDF beats BOW for search and retrieval
If a word appears in EVERY document in the corpus, what is its IDF score?
- TF-IDF = TF(t,d) × IDF(t), where IDF = log(N/DF(t)). Words appearing in every document get IDF ≈ 0 and are automatically suppressed — no manual stopword list needed.
- sklearn's TfidfVectorizer uses smoothing (+1 in numerator and denominator of the log) and L2 row-normalisation — values differ slightly from raw TF-IDF but rank ordering is preserved.
- Cosine similarity over TF-IDF vectors is the backbone of classical document retrieval — semantically similar documents share rare, document-specific terms, which cosine rewards.
- TF-IDF is BOW with smart weights — the matrix shape is identical (n_docs × vocab_size) and all sparse-matrix tricks still apply; only the values change.
Every search engine and document-retrieval system uses TF-IDF as the baseline. Lucene/Elasticsearch's BM25 is a modern refinement of TF-IDF that adds length normalisation and term saturation. Modern dense-vector search systems (FAISS, vector DBs) typically dual-use TF-IDF for sparse retrieval and embeddings for dense retrieval, combining both via reciprocal rank fusion.
If you remove it: Without TF-IDF intuition, BM25 and dense-sparse hybrid retrieval feel arbitrary. TF-IDF is the conceptual stepping stone between raw BOW and modern retrieval architectures.