4 · What are embeddings?
An embedding maps text to a vector so that similar meanings land near each other. Semantics becomes geometry — the foundation of every search, RAG, and recommendation system you'll build.
An embedding is a function that maps a piece of text to a fixed-length vector of numbers, trained so that texts with similar meaning produce nearby vectors. Once meaning is a point in space, 'find related text' becomes 'find nearby vectors' — a geometry problem you can solve with arithmetic.
Without this:
Without embeddings you can only match text by exact keywords; a search for 'car' misses 'automobile', and there is no way to feed a model the 'most relevant' documents for a question — the whole idea behind RAG falls apart.
So far an LLM has been a function over tokens. But there's a second, equally important function in every LLM app: the embedding model. Its job is not to generate text — it is to turn a piece of text into a single vector of numbers, called an embedding.
embed("the cat sat on the mat") → [0.12, -0.44, 0.91, ...] (e.g. 1536 numbers)
What makes an embedding useful is not the individual numbers — it's the geometry of the whole space. The model is trained so that texts with similar meaning produce nearby vectors, and unrelated texts produce far-apart ones. "A dog chased the ball" and "the puppy ran after the toy" land close together even though they share almost no words; "the stock market crashed" lands far away.
This single property — meaning becomes distance — is the engine behind:
- Semantic search: find documents whose embedding is closest to the query's embedding.
- RAG: retrieve the nearest chunks of your knowledge base and paste them into the prompt.
- Clustering / recommendation / deduplication: group or rank by vector proximity.
Real embedding models are trained neural networks (you saw the machinery in the NLP and Deep Learning tracks). But to understand what they give you, we'll build a tiny deterministic one from scratch — a bag-of-words embedder — and show that even this toy version makes related sentences score higher than unrelated ones.
Here is the toy embedder. We fix a small vocabulary (the dimensions of our space), and embed each sentence as a vector whose i-th entry counts how many times vocabulary word i appears. Then we L2-normalize each vector (scale it to length 1) so that long and short sentences are comparable — only direction matters, not how many words there are. Finally we score pairs by dot product: for unit vectors, a bigger dot product means the vectors point in more similar directions, i.e. the sentences share more meaning.
Python (in browser)
A bag-of-words embedder: text becomes a normalized vector, and dot products surface the sentences that share meaning — semantics turned into geometry.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
In production you don't build the embedder — you call one, exactly like the chat API. You send text and get back a list of floats. Read the OpenAI-style call below; the returned vector plugs straight into the geometry we just demonstrated.
A real embedding call: send text, get a fixed-length float vector. It lives in the same 'meaning = distance' space as our toy embedder — just learned, not hand-built.
What is the defining property of a good text-embedding space?
- An embedding maps text to a fixed-length vector; the embedding model is a separate call from the chat model.
- The defining property is geometric: similar meaning → nearby vectors. This turns 'find related text' into 'find nearby vectors'.
- L2-normalizing makes vectors comparable by direction (meaning) rather than magnitude (length), so dot product reflects similarity.
Semantic search, RAG retrieval, recommendation, clustering, and deduplication all start by embedding text and comparing vectors — embeddings are the input to every chapter that follows.
If you remove it: Without embeddings you're stuck with exact keyword matching: synonyms and paraphrases are invisible, and you can't rank documents by relevance to feed an LLM.