2 · Tokens & the context window
Models don't see words or characters — they see tokens, and only a fixed number at a time. Counting tokens is how you reason about cost, truncation, and what fits.
Text is split into tokens (subword chunks), and a model can only attend to a fixed-size context window of them. Token counts drive cost, latency, and whether your prompt + history + retrieved docs even fit.
Without this:
Ignore tokens and you'll silently truncate the user's question, blow your budget on a runaway prompt, or wonder why the model 'forgot' the start of a long conversation.
A model never sees your raw string. First a tokenizer splits it into tokens — subword chunks like "token", "ization", " the" (you met Byte-Pair Encoding in the Deep Learning track). Each token maps to an integer ID, and that is what the model reads.
Two numbers govern everything:
-
Token count of your input. APIs bill per token (input + output) and latency grows with it. A rough rule for English: ~1 token ≈ 4 characters ≈ ¾ of a word. So 1,000 words ≈ ~1,300 tokens.
-
The context window. A model can only look at a fixed number of tokens at once — its context window (e.g. 8k, 128k, 1M). Everything must fit: system prompt + conversation history + retrieved documents + the space reserved for the answer. When you exceed it, something has to be dropped or the call fails.
LLM engineering is, to a large degree, a token budget problem: you have N tokens of context, and you must spend them wisely between instructions, history, and retrieved knowledge.
Python (in browser)
Token budgeting: count the parts, compare to the window, drop the lowest-priority tokens until it fits. The core loop of every chat app.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
In production, count tokens with the model's real tokenizer (tiktoken / the provider's API) — never guess past the window.
Your prompt's system instruction + chat history + retrieved documents add up to more tokens than the context window. What happens?
- Models read tokens (subword chunks), not words; ~1 token ≈ 4 chars ≈ ¾ word for English.
- The context window caps input + output together; everything (system + history + retrieved + answer space) must fit.
- Token counts drive cost and latency — budget them, and truncate the lowest-priority tokens when over.
Every chat app trims history to stay in-window; every RAG system decides how many retrieved chunks fit the budget; pricing dashboards are denominated in tokens.
If you remove it: Without token awareness you ship apps that silently cut off inputs, exceed budgets, and fail unpredictably on long inputs.