7 · Why RAG?
A model only knows what was in its training data. RAG retrieves your facts, drops them into the prompt, and tells the model to answer from THAT — cutting hallucination and unlocking fresh, private knowledge.
Retrieval-Augmented Generation (RAG) is the pattern retrieve → augment → generate: embed the question, fetch the most relevant text from YOUR data, paste it into the prompt, and instruct the model to answer using only that context. The model's weights stay frozen — you change its knowledge by changing what you put in the context.
Without this:
Without RAG the model answers from frozen, generic training data: it confidently invents facts it never saw (hallucination), can't cite your private documents, and goes stale the moment your data changes.
A language model only knows what was baked into its weights during training. That means three hard limits:
- It hallucinates. Asked something it doesn't know, it doesn't say "I don't know" — it predicts the most plausible-sounding next tokens, which is often a confident, wrong answer.
- It can't see your private data. Your company wiki, last week's support tickets, this customer's order history — none of it was in the training set.
- It goes stale. Training stops at a cutoff date; anything newer simply doesn't exist to the model.
Retrieval-Augmented Generation (RAG) fixes all three with one move: instead of asking the model to answer from memory, you fetch the relevant facts first and paste them into the prompt. The pattern is three steps:
- Retrieve — embed the user's question, then search your data (a vector index from Chapter 2) for the most similar chunks of text.
- Augment — build a prompt that contains those retrieved chunks as context, plus an instruction: "answer using only the context above."
- Generate — call the model. Now it answers grounded in your facts instead of from frozen memory.
The model's weights never change. You change what it knows by changing what you put in front of it. That is the single most important LLM-engineering pattern in production.
Here is a complete toy RAG over five facts. We use a deterministic bag-of-words embed() (no real embedding model needed), cosine similarity to retrieve the top-k facts, and a mock generate() that composes an answer by stitching the retrieved facts together. The mock is a stand-in for a real LLM call — but it makes the key point visible: the answer is built only from what retrieval found, not from memory.
Python (in browser)
End-to-end toy RAG: embed → cosine-retrieve top-k → mock generate. The answer is built only from retrieved facts, so it can't drift from your data.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
The production shape is identical to the toy: embed the question, retrieve chunks, paste them into a grounded prompt, then generate.
What is the core mechanism by which RAG reduces hallucination?
- A model only knows its frozen training data; RAG injects fresh, private, relevant facts into the prompt at query time.
- RAG = retrieve (embed + search your data) → augment (build a grounded prompt) → generate (the model answers from the context).
- Most production 'hallucination' bugs are really retrieval bugs — fix what context the model sees, not the model.
Doc-Q&A bots, customer-support assistants, 'chat with your PDF', and AI search all use RAG to ground answers in a specific corpus instead of generic training data.
If you remove it: Drop RAG and your assistant answers from memory — confidently wrong on anything private, recent, or niche, with no way to cite a source.