12 · Word embeddings: dense vector space
Map every word to a dense 50–300 dimensional vector so that semantic similarity becomes geometric proximity — and 'king − man + woman ≈ queen' becomes literally true.
Map every word to a DENSE vector (50–300 dims) such that semantic similarity becomes geometric proximity — and 'king − man + woman ≈ queen' becomes literally true.
Without this:
Stuck in sparse one-hot/BOW/TF-IDF land where every word is orthogonal to every other.
Until now, every word representation you've seen — one-hot encoding, Bag of Words, TF-IDF — shares one structural property: the vocabulary forms the columns of the matrix. A vocabulary of 50,000 words means a 50,000-dimensional vector. Most entries are zero. "cat" and "feline" are orthogonal — cosine similarity exactly 0.0 — even though they mean almost the same thing.
Dense embeddings break this constraint entirely. Instead of indexing into a vocabulary, each word gets a real-valued vector of fixed, small dimensionality — typically 50 to 300 dimensions. Every dimension has a non-zero value. The geometry of the space is learned from data, not imposed by dictionary order.
The remarkable property that emerges: semantic similarity becomes geometric proximity. Words used in similar contexts end up with similar vectors. "dog" and "puppy" land close together. "Tokyo" and "Paris" land close together, far from "banana". And arithmetic in this space captures analogical relationships:
king − man + woman ≈ queen
Paris − France + Italy ≈ Rome
This is not hand-engineered. It falls out of training a neural network to predict context.
The two most famous dense embedding methods — Word2Vec (covered in the next two lessons) and GloVe — learn these vectors from co-occurrence patterns in hundreds of millions of words of text. Pre-trained embeddings are reused constantly: you download the 300-dimension GloVe or Word2Vec weights and immediately have semantically rich word representations without training anything from scratch.
The king − man + woman ≈ queen analogy as vector arithmetic
Python (in browser)
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Sparse vs dense: one-hot, TF-IDF, and embeddings at a glance
Why are embeddings called 'dense' and not sparse?
- Dense embeddings map each word to a low-dimensional real-valued vector (50–300 dims) where ALL dimensions are non-zero — completely eliminating the sparsity of one-hot, BOW, and TF-IDF.
- Semantic similarity becomes geometric proximity: cosine('cat', 'dog') ≈ 0.8 while cosine('cat', 'justice') ≈ 0.1 — learned automatically from co-occurrence patterns, without any hand-crafted rules.
- Analogy arithmetic (king − man + woman ≈ queen) works because linear relationships in the embedding space encode relational structure — but it fails ~20–30% of the time on standard benchmarks, so treat it as an emergent property, not a guarantee.
- Static embeddings (Word2Vec, GloVe, fastText) give one vector per word regardless of context. Contextual embeddings (BERT, GPT) give different vectors for the same word in different sentences — strictly more powerful but computationally heavier.
Word2Vec, GloVe, fastText, and BERT — every modern NLP model either produces or consumes dense embeddings. Vector databases (Pinecone, Weaviate, Qdrant, pgvector) store embedding vectors and retrieve the nearest neighbors at query time. Semantic search, recommendation systems, and RAG (retrieval-augmented generation) pipelines all depend on dense embeddings as their foundational representation.
If you remove it: Without understanding dense embeddings, vector databases, semantic search, and modern LLM architecture all feel like magic. Embeddings are the conceptual bridge between classical NLP (sparse counts) and modern deep learning (everything is a continuous vector).