18 · Kindle sentiment: AvgWord2Vec vs TF-IDF
Build AvgWord2Vec features for binary Kindle-review sentiment and run a head-to-head comparison with TF-IDF — the daily workflow of an applied NLP engineer.
Use AvgWord2Vec features for a binary Kindle-review sentiment task — see when dense embeddings beat sparse counts.
Without this:
Without the head-to-head, dense vs sparse stays abstract.
You have already built a spam pipeline with BOW and a BOW-vs-TF-IDF A/B comparison. This lesson reuses the same pipeline shape on a different task — binary sentiment classification on synthetic Kindle reviews — and swaps one component: instead of a bag-of-words feature matrix, you feed in averaged Word2Vec vectors.
The key engineering insight is that the pipeline skeleton does not change at all:
vectorize → LogisticRegression → evaluate
What changes is what "vectorize" means:
- TF-IDF: a sparse matrix of term frequencies weighted by IDF. Each dimension is a vocabulary token.
- AvgWord2Vec: a dense vector of fixed width (50-D here). Each dimension is a learned semantic axis, averaged across all tokens in the review.
The demo caveat: gensim is not available in Pyodide, so instead of loading actual pre-trained Word2Vec weights we construct a random 50-D embedding matrix indexed by vocabulary. This mimics the pipeline shape without providing real semantic knowledge. On our tiny 40-review dataset with random vectors, TF-IDF typically wins — because random embeddings carry no semantic signal. The pyRead block explains exactly when real pre-trained embeddings (GloVe, fastText) do beat TF-IDF.
Python (in browser)
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
When real pre-trained embeddings beat TF-IDF: small data + synonym sensitivity
Why do REAL avg-Word2Vec features typically beat TF-IDF on SMALL training datasets?
- The AvgWord2Vec pipeline has the same skeleton as TF-IDF plus LogReg: vectorize, classify, evaluate. Only the vectorization step changes — a dense averaged embedding replaces the sparse term-frequency matrix.
- Random embeddings carry zero semantic signal — they are a pipeline-shape demo only. REAL pre-trained embeddings (GloVe, Word2Vec, fastText) encode semantic relationships built from billions of training tokens and are the ones that beat TF-IDF on small datasets.
- Small data means real pre-trained dense features win; large data means sparse TF-IDF can match or beat them. The crossover is roughly 1 000 to 2 000 labeled examples for sentiment tasks.
- Modern best practice for small-data sentiment: sentence-transformers paraphrase-MiniLM-L6-v2. It averages transformer sub-word embeddings into a single 384-D vector per sentence and achieves 85 to 95 percent accuracy with as few as 50 training examples.
Production sentiment systems combine sparse (TF-IDF) and dense (transformer) features via FeatureUnion; the head-to-head comparison done here is the daily life of an applied NLP engineer.
If you remove it: You'd default to TF-IDF without exploring when embeddings help.