14 · Word2Vec: Skip-Gram
Skip-Gram flips CBOW: predict each context word FROM the center word — better for rare words and specialized corpora, slower to train.
Skip-Gram FLIPS CBOW: predict each context word FROM the center word — better for rare words, slower to train.
Without this:
Stuck only on CBOW which biases against rare words.
CBOW predicts the center word from an average of context words. This averaging is computationally efficient but it has a downside: rare words contribute equally to the average but receive few gradient signals back, so their embeddings remain noisy.
Skip-Gram flips the direction: given the center word, predict each surrounding context word individually. For "the quick brown fox jumps" with window=2, the training pairs generated for the center word "brown" are:
(brown, the) (brown, quick) (brown, fox) (brown, jumps)
Each pair is an independent training signal. A rare word that appears even once generates training pairs for every word in its window — giving the embedding far more gradient updates per occurrence than CBOW would.
The downside: with a window of 2 and N center words, Skip-Gram generates ~2 × 2 × N = 4N training pairs, compared to N pairs in CBOW. Training is 3–5× slower.
The rule of thumb: CBOW for large general corpora, Skip-Gram for small specialized corpora with important rare terms.
Skip-Gram architecture: center word → predict each surrounding context word independently
Python (in browser)
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
gensim Skip-Gram (sg=1) — run this locally; gensim is not available in Pyodide
Python (in browser)
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
CBOW vs Skip-Gram decision guide
If your corpus has many rare specialized terms, which do you prefer — CBOW or Skip-Gram?
- Skip-Gram's objective: given the center word, predict each surrounding context word individually. A corpus of N words with window W generates ~2WN training pairs — 3–5× more than CBOW, hence slower training.
- Skip-Gram favors rare words: each occurrence generates W independent training pairs, giving rare tokens far more gradient updates per appearance than CBOW's single averaged update.
- Negative sampling (5–20 random 'wrong' context words per positive pair) replaces full-softmax, reducing compute from O(V) to O(k) per update — the key trick enabling Word2Vec on billion-word corpora.
- Word2Vec embeddings are static: one fixed vector per word regardless of context. This is its main limitation compared to contextual embeddings (BERT, GPT) — but static embeddings are still competitive on tasks with short, unambiguous documents.
Pre-trained Skip-Gram is the basis of fastText and many specialized domain embeddings (biomedical: BioWordVec, legal: LegalBERT's initialization, code: code2vec). The (center, context) pair generation pattern survived into modern contrastive learning (CLIP's (image, caption) pairs, SimCLR's (augmented view, augmented view) pairs) — contrastive learning IS Skip-Gram generalized to arbitrary modalities.
If you remove it: Without Skip-Gram, contrastive learning (CLIP, SimCLR, DPR, E5) is a mystery. All of these methods generate positive pairs (anchor, target) and negative pairs (anchor, random), exactly as Skip-Gram does with (center, context) and (center, noise-word). Understanding Skip-Gram means understanding the family of self-supervised representation learning methods that dominate modern AI.