LoRA: customizing a giant AI without retraining all of it

What LoRA is

LoRA is a cheaper way to fine-tune a large AI model, which means giving it extra training on your own examples so it gets better at your task. Instead of adjusting the whole model, LoRA leaves the original untouched and trains a small add-on that sits beside it. The name is short for Low-Rank Adaptation, and "low-rank" is the trick that keeps the add-on small.

Why the full version is so expensive

Underneath, a model is a huge collection of numbers called parameters, set during its original training. Fine-tuning all of them means the computer has to hold three things for every single parameter: the number itself, its gradient (which way and how much that number should move) and the optimizer state (running notes the training method keeps about each number). Together that comes to about 16 bytes per parameter.

Imagine a model with 10 billion parameters. At about 16 bytes each, that is roughly 160 gigabytes of memory just to hold the numbers and their training bookkeeping. For large models, fine-tuning every parameter is a job for a cluster: many machines working together.

What the video shows

The video frames LoRA as customizing a giant AI with sticky notes. Retraining every part of a huge model is expensive, so this trick freezes the model and trains a tiny add-on beside it, using only a small fraction of the numbers.

An everyday example

Imagine a thick printed manual that a whole company relies on, and one team needs it adapted to their work. Reprinting the entire manual with the changes would be slow and costly. So they put sticky notes on the pages that matter: "at this step, do it this way instead". The manual itself never changes. The notes are small, cheap and easy to peel off, and anyone reading the manual with the notes attached gets the adapted version.

How it works

• Freeze. The original parameters are locked. They are still used to produce answers, but they never change.
• Add a slim pair. Next to a big table of numbers inside the model, LoRA places two thin tables. Multiplied together, they produce a correction the same size as the big table, which is added on top. That is what "low-rank" means here: a big change built from two slim pieces.
• Train only the pair. Only the thin tables learn, so only they need gradients and optimizer notes.

Here is a worked example with made-up round numbers. A table of 4,000 by 4,000 holds 16 million numbers. Two slim tables, one of 4,000 by 8 and one of 8 by 4,000, hold 64,000 numbers between them. That is 250 times fewer, and yet together they still produce a full 4,000 by 4,000 correction. Repeated across a whole model, this is how LoRA cuts the number of trainable parameters by orders of magnitude, meaning a hundredfold or more, not a few percent.

Why it matters

The frozen numbers still have to sit in memory, but they no longer need gradients or optimizer notes. Those are kept only for the tiny add-on. That makes adapting a big model far less demanding, and the original model stays intact, with the new behaviour living in a small, separate piece.

What LoRA does not answer is whether fine-tuning is worth paying for in the first place. That is a separate question, and the course takes it on in Chapter 20, where a LoRA add-on is written by hand and priced. If memory is still too tight, a close cousin called QLoRA also compresses the frozen model.

Learn it step by step in Chapter 11 of our free course AI From Scratch: From Base Model to Assistant: SFT, RLHF, DPO and GRPO.