18 · Building fine-tuning data
Fine-tuning is only as good as its dataset. The format is a list of chat conversations; the work is validation, dedup, and a clean train/val split — exactly the boring, decisive part.
A fine-tuning dataset is a list of CONVERSATIONS, each a list of {role, content} messages. The model learns to produce the assistant turn given everything before it. The engineering is not the training call — it's validating the format, removing duplicates, and splitting train/val so your metric isn't a lie.
Without this:
Skip validation and dedup and you train on malformed rows and duplicates that leak into your validation set, so the model learns noise and your eval score lies about how good it is.
Fine-tuning a chat model means showing it many conversations and letting it learn to produce the assistant's reply. The de-facto format (used by OpenAI, and close to what transformers chat templates expect) is one JSON object per example, each with a messages list:
{"messages": [
{"role": "system", "content": "You are a terse support bot."},
{"role": "user", "content": "How do I reset my password?"},
{"role": "assistant", "content": "Open Settings -> Security -> Reset password."}
]}
The model is trained to predict the assistant turn given the system + user turns before it. So your dataset is literally a pile of "here is the situation, here is the ideal answer" pairs in chat clothing.
Now the part that actually decides whether fine-tuning works — and it is not glamorous:
- Format validation. Every example needs a non-empty
messageslist, validrolevalues, non-emptycontent, and at least oneassistantturn (otherwise there's nothing to learn from). Malformed rows either crash the job or silently teach garbage. - Deduplication. Duplicate examples over-weight whatever they say and — worse — the same example can land in both train and validation, so your eval score is inflated by memorization. Dedup by normalized content.
- Train/val split. Hold out a slice the model never trains on, so you can measure whether it generalizes. The split must be deterministic (seeded) so runs are comparable, and dedup must happen before the split so no example crosses the boundary.
Quality beats quantity here: a few hundred clean, consistent, deduplicated examples routinely outperform thousands of noisy ones.
Below is the full pipeline on a tiny dataset. It deliberately contains a malformed example (no assistant turn), an empty-content example, and a duplicate, so you can watch each guard fire. Read the printed report: it tells you exactly which rows were rejected and why, then dedups, then makes a seeded, hash-based split so the same example always lands on the same side — no leakage.
Python (in browser)
The real fine-tuning work: validate every conversation, drop duplicates and malformed rows, then split deterministically so no example leaks across train/val.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
One quality dimension the code can't fully catch is label consistency: if two near-identical user questions get contradictory assistant answers in your dataset, you're teaching the model to be inconsistent. A cheap automated check is to bucket examples by their (normalized) user turn and flag any bucket whose assistant answers disagree — surfacing the rows a human must reconcile before training.
The provider's training call is one line; the dataset you feed it is the whole job. Validate, dedup, and split before this point — the API won't fix your data.
Why must you deduplicate the dataset BEFORE the train/validation split?
- Fine-tuning data is a list of conversations: each example is a {messages:[{role,content}...]} object, and the model learns to produce the assistant turn.
- Validate every row (non-empty messages, valid roles, non-empty content, at least one assistant turn) and reject the rest — garbage in, garbage learned.
- Dedup by normalized content, THEN make a deterministic (seeded/hash) train/val split so no example leaks and your metric is honest.
Every fine-tuned model — from house-voice chatbots to JSON-emitting classifiers — is produced from a JSONL of chat conversations that someone validated, deduped, and split.
If you remove it: Skip the data hygiene and you train on malformed rows and leak duplicates into validation, so the model learns noise and your eval score lies.