11 · Structured (JSON) output
To use a model's output in code you need structure, not prose. Ask for JSON, extract it, parse it, validate it against a schema — and repair or retry when the model returns something malformed.
When code consumes a model's output, free-form text is a liability — you need JSON that matches a schema. The robust pattern is: ask for JSON, EXTRACT the JSON substring (models add stray prose), json.loads it, VALIDATE required keys and types, and on failure REPAIR what you can or RETRY. Never trust the raw string.
Without this:
Without parse-and-validate you'll crash on the day the model wraps its JSON in 'Sure, here you go:' or drops a required field. With it, your pipeline degrades gracefully — repair, retry, or reject — instead of throwing in production.
A chatbot can answer in prose, but an LLM app usually needs the output in code — to store it, branch on it, or call another function. That means the model must return structured data, almost always JSON, that matches a fixed schema (which keys must exist, what type each value is).
The catch: a language model emits text. Even when you ask for JSON, it may add a friendly preamble ("Sure! Here's the JSON:"), wrap it in Markdown code fences, omit a required field, or produce a trailing comma that breaks the parser. So the consumer side needs a small, defensive pipeline:
- Extract the JSON substring. Models love to surround JSON with prose — slice from the first
{to the matching last}instead of feeding the whole reply to the parser. - Parse with
json.loads. If it raises, you have malformed JSON — don't crash, handle it. - Validate against the schema: every required key present, every value the right type. Parsing success is not the same as a valid object.
- Repair or retry on failure. Some errors you can fix locally (strip fences, coerce a number written as a string); others you bounce back to the model with the error message and ask it to try again.
This "ask → extract → parse → validate → repair/retry" loop is the backbone of every reliable LLM feature that feeds a database or another service.
Below, four simulated model outputs stand in for what a real model returns — one clean JSON object, one wrapped in chatty prose, one inside a Markdown code fence, and one genuinely malformed (a trailing comma). We run each through extract → parse → validate against a schema, then apply a tiny repair step and re-validate. Watch which cases pass on the first try and which only pass after repair.
Python (in browser)
Extract → parse → validate → repair → re-validate on four simulated replies: clean passes instantly; fenced/trailing-comma cases only pass after a cheap local repair.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Some failures can't be fixed locally — if the model omits a required key, no string surgery will conjure it. The honest move is to retry: send the request again with the validation error appended, so the model sees what went wrong and corrects it. Below is a deterministic mock model (a stand-in) that returns bad JSON on its first attempt and good JSON on the second, wrapped in a retry loop with a hard cap so you never loop forever.
Python (in browser)
Retry-with-feedback: append the validation error and re-ask. The mock model fails once, then self-corrects — bounded by max_retries so it can't loop forever.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Real APIs offer schema-constrained decoding (OpenAI response_format json_schema) so the model can't drift from your shape — but defensive parse + validate is still your safety net.
A model returns the string '```json\n{"label": "bug"}\n```'. Your code does json.loads on the raw reply and it raises a JSONDecodeError. What is the correct fix?
- Apps need structured data (JSON matching a schema), not prose — the consumer side must extract, parse, validate, and repair/retry.
- Models wrap JSON in prose or code fences: slice from the first { to the last } before json.loads instead of feeding the whole reply.
- Parsing success ≠ valid object: always check required keys AND value types; repair locally when cheap, retry with the error fed back when not (bounded by max_retries).
Tool/function calling, data-extraction features, and agent steps all depend on schema-valid JSON; libraries like Pydantic, Instructor, and OpenAI's response_format json_schema automate exactly this extract-parse-validate loop.
If you remove it: Skip validation and your app crashes or silently stores garbage the first time the model adds a preamble, omits a key, or returns the wrong type — failures that only surface in production.