40 · Transfer learning & fine-tuning
Pretrain once (expensive), adapt many times (cheap). Feature extraction freezes the backbone and trains a tiny head; full fine-tuning updates every weight — each suits a different data regime.
Transfer learning reuses a pretrained backbone: either FREEZE it and train only a small new head (feature extraction) or update ALL weights (full fine-tuning). The data regime decides which.
Without this:
Nobody trains LLMs from scratch per task — that costs millions. Transfer learning is the everyday workflow: grab a pretrained model and adapt it cheaply to your problem.
Lesson 38 set up the key asymmetry: pretraining is expensive (trillions of tokens, GPU-months, millions of dollars), but adapting a pretrained model to a specific task is cheap. Transfer learning is the workflow that exploits this — pretrain once, adapt many times. You never train the giant model from scratch for your problem; you start from a shared pretrained backbone and adjust it.
There are two main strategies, and the choice depends on how much labeled data you have:
-
Feature extraction (freeze the backbone). Keep all the pretrained weights FROZEN — treat them as a fixed feature extractor — and train only a small new head on top (for example, a linear classifier mapping the backbone's output to your classes). Only the head's handful of parameters are updated. This is fast, memory-light, and resistant to overfitting because the powerful pretrained features can't be damaged.
-
Full fine-tuning (update all weights). Unfreeze the entire model and let gradients flow through ALL of it, gently nudging every pretrained weight toward your task. This can squeeze out the best accuracy when you have enough labeled data, but it updates hundreds of millions of parameters and risks overfitting (and "catastrophic forgetting") on small datasets.
The mental model: feature extraction reuses the pretrained knowledge as-is and only learns how to READ it for your task; full fine-tuning re-shapes the knowledge itself. The cell below makes the parameter-count difference concrete — it is enormous.
Python (in browser)
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
HuggingFace workflow: AutoModelForSequenceClassification.from_pretrained, optional requires_grad=False to freeze, Trainer.train() with a small LR (2e-5) and warmup.
When should you freeze the backbone and train only a new head?
- Transfer learning reuses a pretrained backbone instead of training from scratch: pretrain once (expensive), adapt many times (cheap).
- Feature extraction freezes the backbone and trains only a tiny head (often <0.01% of params) — fast and overfitting-resistant; full fine-tuning updates all weights for higher accuracy when data is plentiful.
- Full fine-tuning on small data risks overfitting and catastrophic forgetting; fine-tune with a small learning rate (e.g. 2e-5) and warmup to preserve pretrained knowledge.
Almost every applied NLP/vision project is a transfer-learning project: download a pretrained model from the HuggingFace Hub and fine-tune (or feature-extract) it on your data.
If you remove it: You'd think every new task needs a model trained from scratch — and either burn enormous compute or, on small data, fail entirely without the pretrained head-start.