13 · Word2Vec: CBOW (Continuous Bag of Words)
CBOW trains a shallow neural net to predict a center word from its surrounding context — the hidden-layer weights ARE the embeddings.
CBOW trains a shallow neural net to PREDICT a center word from its surrounding context — the hidden-layer weights ARE the embeddings.
Without this:
Word2Vec stays an opaque library call instead of an algorithm you understand.
Word2Vec is not one model — it is a family of training objectives that all produce the same type of output: a matrix of learned word vectors. The two variants are CBOW (Continuous Bag of Words) and Skip-Gram (next lesson).
CBOW's learning objective is simple: given the surrounding context words, predict the center word. For the sentence "the quick brown fox jumps" with a window of 2, the training example for "brown" is:
context: ["the", "quick", "fox", "jumps"] → predict: "brown"
The network architecture is intentionally shallow — just two weight matrices:
- Embedding matrix (V × D): one row per vocabulary word; this is what we want to learn.
- Context matrix (D × V): the output projection; often discarded after training.
The four context words are one-hot encoded, mapped through the embedding matrix (one row lookup per word), averaged to produce a single D-dimensional vector, projected through the context matrix, and passed through a softmax to produce a probability distribution over the entire vocabulary.
The "fake task" of predicting the center word forces the network to compress co-occurrence statistics into the D-dimensional hidden layer. After training on millions of sentences, that hidden layer happens to encode semantic relationships. The weights are the embeddings.
CBOW architecture: one-hot inputs → embedding lookup → average → predict center word
Python (in browser)
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
CBOW vs Skip-Gram: direction of prediction, speed, and bias toward rare words
In CBOW, the hidden layer weights store...
- CBOW's objective is: given surrounding context words (window ±2), predict the center word. The shallow network has two weight matrices: W1 (the embeddings) and W2 (the output projection, discarded after training).
- Negative sampling replaces full softmax over the vocabulary: instead of normalizing over all V words, sample 5–20 'negative' wrong words and solve a binary classification problem per sample — the key efficiency trick that made Word2Vec practical.
- SVD of a co-occurrence matrix closely approximates CBOW — GloVe makes this connection explicit by fitting directly on the log co-occurrence matrix. Both methods extract the same type of semantic structure.
- CBOW averages context → biased toward frequent words, fast to train; prefer it for large general corpora. Skip-Gram predicts each context word individually → better for rare words, slower; prefer it for specialized domains.
Pre-trained Word2Vec and GloVe embeddings are still used as the first layer of many text classifiers and NER models. The Word2Vec 'predict context to learn representations' idea generalised to item2vec (recommender systems), node2vec (graph embeddings), and protein embeddings in bioinformatics — any sequence where co-occurrence is meaningful can be embedded with this approach.
If you remove it: Without understanding CBOW, the self-supervised pre-training idea behind BERT and GPT is a black box. BERT's masked language model and GPT's next-token prediction are both 'fake task' self-supervised objectives that produce contextual embeddings — they are Word2Vec with transformers replacing the shallow network.