3 · Sampling & decoding
The model outputs probabilities; YOU choose how to turn them into a token. Temperature, top-k, and top-p are the knobs that control creativity vs reliability.
The model gives you a probability distribution; the decoding strategy turns it into a token. Temperature sharpens or flattens it, top-k keeps the k best options, and top-p (nucleus) keeps the smallest set covering probability p — trading determinism for diversity.
Without this:
Without controlling decoding you get either robotic repetition (temperature too low) or incoherent rambling (too high), and you can't make extraction tasks reproducible.
The model's raw output is a vector of logits — one real number per vocabulary token. Two steps turn that into a chosen token:
- Softmax converts logits into a probability distribution.
- A decoding strategy picks one token from that distribution.
The decoding strategy is yours to choose, and it dramatically changes behaviour:
- Greedy — always take the single most likely token. Deterministic, but tends to get repetitive and bland.
- Temperature — divide the logits by a temperature
Tbefore softmax.T<1sharpens the distribution (more confident, more deterministic);T>1flattens it (more random, more creative);T→0approaches greedy. - Top-k — keep only the
khighest-probability tokens, renormalize, and sample among them. Caps how weird the choice can get. - Top-p (nucleus) — keep the smallest set of tokens whose probabilities sum to
p(e.g. 0.9), renormalize, sample. Adapts the cutoff to how confident the model is.
Rule of thumb: low temperature (0–0.3) for extraction, classification, code, JSON (you want the same answer every time); higher (0.7–1.0) for brainstorming and creative writing.
Python (in browser)
Temperature reshapes the SAME logits: low T → sharp/deterministic, high T → flat/diverse.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Now the full decoders. Watch how greedy is fully deterministic, while top-k and top-p restrict which tokens can be sampled before adding randomness — the standard way to keep generation coherent without being robotic.
Python (in browser)
Greedy, temperature, top-k and top-p side by side — the four decoders behind every API's `temperature` / `top_p` parameters.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
You're building a feature that extracts a person's name and email from text into JSON. Which decoding setting is most appropriate?
- Softmax turns logits into probabilities; the decoding strategy turns probabilities into a token — and you choose it.
- Temperature sharpens (T<1) or flattens (T>1) the distribution; top-k caps the candidate count; top-p (nucleus) keeps the smallest set covering probability p.
- Low temperature for extraction/code/JSON (reproducible); higher for brainstorming/creative writing.
Every API's `temperature` and `top_p` parameters are exactly these knobs; agent frameworks set temperature 0 for tool-calling so the plan is stable.
If you remove it: Without decoding control you can't trade off reliability vs creativity, and structured outputs become non-reproducible.