L1 vs L2 regularization: shrink every setting, or switch some off

What L1 and L2 regularization are

An AI model makes its predictions with many internal numbers called weights. Each weight says how strongly one input, or one internal signal, should count. When some weights grow very large, the model overreacts: a small change in the input can swing the output wildly.

Regularization means adding a charge for large weights to the score that training tries to lower. That score is called the loss, and it measures how wrong the model is. With the extra charge, the model has two goals at once: fit the examples, and keep its weights modest. L1 and L2 are the two classic ways to measure "too big":

• L2 charges for each weight squared. It is also known as weight decay.
• L1 charges for each weight's plain size, ignoring whether it is positive or negative.

What the video shows

The video presents both as ways to calm a model that overreacts, by penalising big settings. L2 shrinks all of them a little. L1 switches the unneeded ones off completely, setting them to exactly zero.

Why they behave so differently

The difference comes from how hard each penalty pulls as a weight gets smaller.

• L2 pulls in proportion to the weight. A big weight gets a hard tug; a small one only a gentle one. As the weight shrinks, the pull fades, so it creeps toward zero but rarely lands on it. Think of a spring: the closer it gets to its resting point, the weaker it tugs.
• L1 pulls with the same strength all the way down. A weight that is not earning its place gets pushed the whole way to zero and stays there. Think of a steady wind that does not ease off as you approach the wall.

So L2 leaves you with many small weights and a smoother, calmer model. L1 leaves many weights at exactly zero, which means the inputs attached to them are simply ignored. In effect, L1 chooses which inputs (also called features) matter.

An everyday example

Imagine trimming a monthly budget with 30 spending lines, a few of them small subscriptions you never use. L2 is like cutting every line by 10%: the big lines lose the most and the total drops, but no line ever reaches zero, so you still pay a little for apps you never open. L1 is like cutting the same flat $5 from every line: the small, unused subscriptions hit zero and are cancelled, while the big lines you rely on only get slightly smaller.

It is the spring and the wind again: a percentage cut gets smaller as the line shrinks, so it never finishes the job, while a flat cut stays the same size and does.

When you would use each

| | L2 | L1 | |---|---|---| | What it does to weights | Makes them small | Makes many exactly zero | | What you get | A smoother model | A sparse model (mostly zeros) that uses fewer inputs | | Pick it when | You want steadier predictions | You suspect many inputs are useless |

Why it matters

In our course's experiment, a very flexible curve that passed through all twelve of its training points went wild on new data, and its single largest weight did most of the damage. Charging the model for big weights is a direct cure: no single number gets to dominate. That is why L1 and L2 sit among the standard tools against overfitting (memorising the training examples instead of learning the pattern), alongside dropout, early stopping and making extra training examples from the ones you have.

Learn it step by step in Chapter 6 of our free course AI From Scratch: Getting It to Train, and Getting It to Generalise.