19 · LoRA in one cell
Full fine-tuning updates every weight — billions of them. LoRA freezes the original weights and learns a tiny low-rank update B·A instead, cutting trainable parameters by orders of magnitude with almost no loss.
LoRA (Low-Rank Adaptation) freezes the pretrained weight W and learns a low-rank update ΔW = B·A, where B is d×r and A is r×d with rank r tiny (e.g. 8). The effective weight is W + (α/r)·B·A. You train only 2·d·r numbers instead of d·d — a reduction of orders of magnitude.
Without this:
Without parameter-efficient fine-tuning, adapting a billion-parameter model means storing and updating a full copy of every weight per task — infeasible on consumer hardware and wasteful when one adapter per task would do.
Full fine-tuning re-trains every weight in the model. For a layer whose weight matrix W is d × d, that's d² trainable parameters — and a real model has hundreds of such matrices, totaling billions. You also have to store a whole new copy of the model per task. That's the wall LoRA was built to climb.
The key observation behind LoRA (Low-Rank Adaptation): the update you need to specialize a pretrained model usually has very low intrinsic rank — it can be well-approximated by the product of two skinny matrices. So instead of learning a full d × d update ΔW, LoRA:
- Freezes the original weight
W(no gradients, never changes). - Learns two small matrices:
Bof shaped × randAof shaper × d, where the rankris tiny (often 4, 8, 16). - Uses the effective weight
W + (α/r)·B·Aat inference, whereαis a scaling constant.
The update ΔW = B·A is a full d × d matrix, but it's built from only d·r + r·d = 2·d·r trainable numbers instead of d·d. When r ≪ d that's a massive saving. At training time you back-prop only into A and B; W sits frozen. At deploy time you can either keep the adapter separate (swap adapters per task) or fold (α/r)·B·A into W so inference costs nothing extra.
This is exactly the low-rank / matrix-factorization idea from the math track — a big matrix approximated by the product of two thin ones — applied to weight updates.
Here is LoRA in one NumPy cell. We take a 512 × 512 weight W, build a low-rank update from B (512×r) and A (r×512), form the effective weight W + (α/r)·B·A, and — the headline — count the trainable parameters for full fine-tuning versus LoRA at several ranks. Watch the percentage of parameters you actually train collapse toward a fraction of a percent.
Python (in browser)
LoRA in NumPy: a full d×d update ΔW=(α/r)·B·A is built from only 2·d·r trainable numbers. At rank 8 on a 512×512 layer you train ~3% of the parameters; the matrix interface is unchanged.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
The savings compound at real scale. A 7-billion-parameter model has thousands of weight matrices; full fine-tuning means a 7B-parameter optimizer state and a 7B-parameter checkpoint per task. With LoRA at rank 8 you train and ship a few million parameters — small enough to keep dozens of task adapters on disk and hot-swap them onto one frozen base model. That is what peft does in practice; read the real config below.
PEFT's LoraConfig is exactly our NumPy demo at scale: rank r, alpha scaling, and a list of which weight matrices get a B·A adapter — training well under 0.1% of the parameters.
In LoRA, for a d×d weight matrix with rank r, how many parameters are actually trained, and what stays frozen?
- LoRA freezes the pretrained weight W and learns a low-rank update ΔW = B·A (B is d×r, A is r×d) with tiny rank r.
- You train 2·d·r parameters instead of d·d — orders of magnitude fewer; the effective weight is W + (α/r)·B·A.
- B inits to zero so the adapter starts as a no-op (stable training), and one frozen base model can hot-swap many small task adapters.
LoRA and its quantized variant QLoRA are how almost every open-weight model is fine-tuned today — PEFT's LoraConfig trains under 1% of a 7B model so it fits on a single consumer GPU.
If you remove it: Without parameter-efficient adapters, every task needs a full copy of the model's billions of weights trained and stored — infeasible on commodity hardware and wasteful at scale.