34 · Generators with yield
Write lazy sequences with yield — a function that produces values on demand, uses constant memory regardless of sequence length, and composes cleanly.
A **generator function** (any `def` with `yield`) returns a **generator object** when called — it does **not** run the body immediately. Each call to `next()` on the generator resumes execution from where `yield` last paused it, emitting one value. When the function returns (or falls off the end) Python automatically raises `StopIteration`. The generator is an iterator — it plugs into `for`, `zip`, `list()`, and everything else.
Without this:
Without generators you'd build giant lists to represent sequences — a million-element range would consume millions of bytes before your loop even starts. Generators allow streaming computation: the next value is computed only when asked for, which is the foundation of all memory-efficient data pipelines.
Generators exist to solve two problems simultaneously:
- Memory — if you're producing a long sequence, there's no reason to hold all values in RAM at once. A generator yields one value, discards it, computes the next.
- Composition — generators chain together without intermediate allocations.
gen_b(gen_a(source))pulls values through a pipeline of transformations one at a time.
The syntax is simple: put yield anywhere inside a def and the function becomes a generator function. Calling it returns a generator object; the body hasn't run yet. Every next() call resumes the body until the next yield, then suspends again. There are three ways to write generators in Python: generator functions (def + yield), generator expressions ((...)), and yield from for delegation.
Python (in browser)
Calling `counts_up_to(3)` does not execute a single line of the body — the `for i in range(n)` hasn't started. Only when `next(g)` is called does execution enter the function, run until `yield i`, emit the value, and then freeze.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
The generator expression `(x*x for x in range(N))` creates a tiny object (around 200 bytes) regardless of how big `N` is. The equivalent list comprehension allocates memory for every element upfront. We use 1M here so the cell runs comfortably in your browser; the same generator with `range(1_000_000_000)` would be just as small. For sequences you only partially consume, the generator is always the right choice.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
`yield from f` delegates iteration to the file object itself — each `next()` call reads exactly one line from disk without buffering the whole file. This is the idiomatic pattern for streaming file processing in Python: wrap the open/read logic in a generator and callers never see file handles.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
`yield from iterable` is shorthand for `for x in iterable: yield x` but also correctly threads `send()` and `throw()` calls through the chain (advanced usage). The pipeline pattern here is the Python equivalent of Unix pipes: each stage is independent, composable, and memory-constant.
Does calling a generator function immediately execute its body?
- A generator function is any `def` containing `yield`. Calling it returns a generator object — the body does not run until `next()` is called.
- Generator expressions `(expr for x in iterable)` are the one-liner equivalent of generator functions and use constant memory.
- `yield from iterable` delegates to another iterable, making it easy to compose generators into pipelines without intermediate allocations.
- A 10M-element generator expression occupies ~200 bytes; the equivalent list occupies ~80 MB. For sequences consumed once or partially, prefer the generator.
Streaming datasets that don't fit in RAM are modelled as generators: `def batches(path, batch_size): ... yield batch` is the pattern behind PyTorch `IterableDataset` and TensorFlow `tf.data.Dataset`. HuggingFace `datasets.IterableDataset` is a generator-based API for datasets too large to cache locally. Even NumPy's `np.load(..., mmap_mode='r')` returns a memory-mapped array that acts like a lazy generator over disk-resident data.
If you remove it: Without generators, loading a 100 GB dataset for fine-tuning would require materialising the full contents in RAM before training can begin — an impossible constraint on most machines. The generator model is what makes it feasible to train on data that is orders of magnitude larger than available memory.