15 · Document vectors via averaged Word2Vec
Average the word vectors of a document to get a single document vector — the simplest sentence embedding that often beats TF-IDF on small corpora.
Average the word vectors of a document to get a single document vector — the simplest 'sentence embedding' that often beats TF-IDF.
Without this:
Word-level embeddings are stuck; you need a DOCUMENT-level representation to feed into a classifier.
Word embeddings are word-level representations. A classifier needs a document-level vector — a single fixed-size vector for each review, email, or article. How do you go from a variable-length sequence of word vectors to one vector?
The simplest approach: average all the word vectors in the document.
def avg_word_vec(tokens, embeddings, vocab_index):
vecs = [embeddings[vocab_index[t]] for t in tokens if t in vocab_index]
return np.mean(vecs, axis=0) if vecs else np.zeros(embedding_dim)
That's it. Zip through the tokens, look up each word's row in the embedding matrix, stack the rows, take the column-wise mean. The result is a single vector of the same dimensionality as the word embeddings (e.g. 300-D), regardless of how many words the document contained.
This averaged vector captures the semantic center of mass of the document. A review about "great food and friendly service" will land in a different region of the 300-D space than a review about "terrible quality and broken product".
Why does this often beat TF-IDF? TF-IDF is a sparse, exact-match representation — if the test document uses "poor" but the training corpus used "terrible", TF-IDF sees no overlap. Averaged Word2Vec is a dense, semantic representation — "poor" and "terrible" have similar vectors (they appear in similar contexts), so the document vectors end up close even if the exact tokens differ.
Python (in browser)
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
avg-Word2Vec: strengths and failure modes
Modern alternatives: SIF, Doc2Vec, Sentence-Transformers (SBERT)
Why does averaging word vectors lose information?
- avg-Word2Vec collapses a variable-length document into a fixed-size vector: look up each token's embedding row, stack, take the column-wise mean. Missing tokens are silently dropped, so handle OOV gracefully with a zero fallback.
- Averaging loses word order and negation — 'dog bites man' and 'man bites dog' produce identical vectors. Use this technique for tasks where bag-of-semantics is sufficient (topic classification, intent detection, rough similarity).
- avg-Word2Vec beats TF-IDF when vocabulary mismatch exists between train and test — semantically similar words have close vectors, so the document vectors stay close even if exact tokens differ.
- Modern upgrade path: SIF weighting for slight improvement; sentence-transformers (`paraphrase-MiniLM-L6-v2`) for state-of-the-art quality with minimal code change.
avg-Word2Vec was the standard document-embedding method in pre-transformer NLP pipelines (2014–2018). It still appears in production systems where inference latency is critical: a single numpy mean is orders of magnitude faster than a transformer forward pass. Semantic search before vector DBs, cluster analysis of survey responses, and document deduplication all used avg-Word2Vec as the representation layer.
If you remove it: Without avg-Word2Vec, sentence-transformers and pooled BERT outputs feel arbitrary. They are all variations on 'collapse token vectors to a single document vector' — avg-Word2Vec is the simplest version of this pattern. Understanding it demystifies [CLS] pooling, mean-pooling, and max-pooling strategies in modern transformer encoders.