20 · Evaluating LLM apps
You can't ship what you can't measure. Exact-match for deterministic tasks, recall@k for retrieval, and groundedness for generation turn 'it feels good' into numbers you can track.
Evaluation makes an LLM app shippable: deterministic tasks use exact-match / contains, retrieval uses recall@k, and generation uses groundedness (is the answer supported by the retrieved context). LLM-as-judge scales subjective grading by asking a strong model to score outputs against a rubric.
Without this:
Without metrics you change a prompt, 'feel' that it got better, and silently regress on the cases you didn't look at. With an eval set every change becomes a number you can defend.
An LLM app is a product, and products need a test suite. But unlike normal code, the output is text, so "did it pass?" needs a metric. The trick is to pick the right metric per task type — there is no single number that captures everything.
1. Deterministic tasks → exact-match / contains. Classification, extraction, math, code: there is a correct answer. Compare the model's output to a gold label with exact string match (normalized: lowercased, stripped) or a looser contains check. This gives you a plain accuracy you can chart.
2. Retrieval → recall@k. Before generation, RAG retrieves the top-k chunks. The retrieval metric asks: of the documents that should have been found (the relevant set), how many appear in the top-k? That fraction is recall@k. If recall@k is low, no amount of prompt-tuning will fix the answer — the model never saw the right text.
3. Generation → groundedness / faithfulness. Even with the right context, the model can hallucinate — assert things the context doesn't support. Groundedness measures the fraction of the answer's content that is actually backed by the retrieved context. A simple proxy is token overlap: what share of the answer's content-words appear in the context? Real systems use an entailment model or an LLM judge, but the idea is the same — penalize claims the context can't justify.
4. LLM-as-judge. For subjective qualities (helpfulness, tone, completeness) you ask a strong model: "Given this question, context, and answer, score faithfulness 1–5 and explain." It scales human judgment to thousands of cases. We describe it here (it needs an API call) but won't run it.
Let's compute the two metrics that decide whether a RAG answer is trustworthy: recall@k for retrieval and a token-overlap groundedness score for generation. We run them on two example cases — one where the answer sticks to the retrieved context (grounded) and one where the model made something up (hallucinated). Watch how groundedness cleanly separates the two even though both sound fluent.
Python (in browser)
recall@k climbs as you widen the retrieval window; token-overlap groundedness cleanly separates a faithful answer from a hallucinated one.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Production eval libraries (Ragas, TruLens, OpenAI evals) replace our token-overlap proxy with an LLM judge, but measure the same thing: is each claim supported by the context?
Your RAG app gives a wrong answer. You measure recall@5 = 0.2 (the relevant chunk usually isn't retrieved). Where should you focus first?
- Pick the metric per task: exact-match/contains for deterministic tasks, recall@k for retrieval, groundedness for generation.
- recall@k = (relevant docs found in top-k) / (all relevant docs); low recall means fix the retriever before the prompt.
- Groundedness/faithfulness measures how much of the answer the context actually supports — the front line against hallucination. LLM-as-judge scales subjective grading.
Every serious RAG deployment tracks recall@k dashboards and faithfulness scores in CI; eval libraries (Ragas, TruLens, OpenAI evals) and LLM-as-judge gate prompt and model changes.
If you remove it: Without evaluation you optimize on vibes, can't catch regressions, and ship hallucinations you never measured.