41 · LoRA & parameter-efficient fine-tuning
Full fine-tuning of a 7B model needs enormous memory. PEFT trains a tiny fraction instead. LoRA freezes the pretrained weight and learns a low-rank update ΔW = B·A — ~64x fewer trainable parameters.
LoRA freezes the pretrained weight W and learns a low-rank update ΔW = B·A (with rank r ≪ d), so only the tiny B and A factors are trained — the effective weight at inference is W + B·A.
Without this:
Full fine-tuning of a multi-billion-parameter model needs optimizer states and gradients for EVERY weight — tens of gigabytes of GPU memory. Without PEFT, adapting large LLMs is out of reach for most people.
Lesson 40's warning has a deeper cost: full fine-tuning of a large model is brutally memory-hungry. Updating a 7-billion-parameter model means storing not just the weights, but also a gradient for every weight AND the optimizer states (Adam keeps two extra values per parameter — momentum and variance). That is several copies of a 7B model in GPU memory at once, tens of gigabytes — far beyond a single consumer GPU.
Parameter-efficient fine-tuning (PEFT) sidesteps this: keep the giant pretrained model frozen and train only a tiny set of NEW parameters. Far fewer gradients and optimizer states to store, so adaptation fits on modest hardware.
The dominant PEFT method is LoRA (Low-Rank Adaptation). Take a pretrained weight matrix W of shape d×d and FREEZE it. Instead of editing W directly, learn a separate low-rank update ΔW = B · A, where B is d×r and A is r×d, with the rank r MUCH smaller than d (e.g. r=8, d=1024). The effective weight used at inference is simply W + B·A. Only B and A are trained; W never changes.
Why is this so much cheaper? A full d×d matrix has d² parameters; the two LoRA factors together have only 2·d·r. With d=1024 and r=8 that is 16,384 trainable parameters instead of 1,048,576 — about 64x fewer. A standard trick: initialize B = 0 so that ΔW = 0 at the start, meaning the adapter begins as a perfect no-op and the model is identical to the frozen base until training nudges B and A away from zero. The cell below makes the shapes, the parameter savings, and the low-rank structure concrete.
Python (in browser)
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
PEFT families: adapters (bottleneck modules), prefix/prompt tuning (virtual tokens), QLoRA (4-bit frozen base + LoRA). The peft library: LoraConfig(r, lora_alpha, target_modules) + get_peft_model.
In LoRA with d=1024 and rank r=8, how many trainable parameters replace the full d×d weight matrix?
- Full fine-tuning of a large model stores gradients + optimizer states for EVERY weight (tens of GB); PEFT freezes the base and trains only a tiny set of new parameters.
- LoRA freezes W and learns a low-rank update ΔW = B·A (B is d×r, A is r×d, r ≪ d); only 2·d·r parameters train (~64x fewer for d=1024, r=8), with effective weight W + B·A. Init B=0 so the adapter starts as a no-op.
- Fine-tuning updates are intrinsically low-rank, so a tiny B·A recovers most of the benefit — and one frozen base can host many swappable task adapters of just a few MB each (QLoRA adds a 4-bit base to shrink memory further).
Almost every modern LLM fine-tune uses LoRA/QLoRA via the peft library — it's how individuals and small teams adapt 7B–70B models on a single GPU, and how providers serve many custom adapters over one shared base.
If you remove it: You'd believe adapting a large LLM always means full fine-tuning and a GPU cluster, missing the cheap, swappable adaptation that makes custom models practical.