Loss curves: what the shape of a training line tells you

What a loss curve is

While an AI model trains, it keeps score of how wrong its answers are. That score is called the loss: one number, where lower means better. Training is the slow process of nudging the model's internal settings, step after step, to push that number down. Plot the loss after every step and you get a line, the loss curve. It is one of the first things worth looking at when a training run misbehaves, because it costs almost nothing to draw.

What the video shows

The video walks through the shapes that line can take: flat, spiking, splitting apart, or dropping too fast. Each shape points to a different problem to check first. The key word is "first". A shape tells you where to start looking. It is not a final diagnosis.

An everyday example

Imagine a doctor reading a patient's temperature chart. A fever of 39 degrees means one thing if it has been creeping up for three days and something else if it appeared within an hour. The chart does not replace the tests, but it decides which test to run first. A loss curve plays exactly that role for an AI.

How to read the shapes

| What the line does | Most likely cause | What to check first | |---|---|---| | Flat from the very first step | The signal inside the model has died | Whether each layer's outputs have collapsed | | Turns into NaN within a few steps | Numbers blew up, or the maths tried to take the log of zero, which has no answer | How big each correction is before it is applied | | Falls, then suddenly jumps | Steps set too big, or one bad batch of examples | Capping the size of each correction; the loss of each batch | | Training loss falls, validation loss rises | Overfitting: memorising, not learning | Penalties that discourage memorising (regularisation), stopping earlier, more data | | Falls suspiciously fast | A leak: the answer is hiding in the inputs, or the test reuses training examples | How the data was split, before celebrating |

A few words in plain English. NaN means "not a number": the computer's way of saying the arithmetic broke. A batch is the small group of examples the model looks at in one step. Validation loss is the same score measured on examples the model is never allowed to learn from, which is how you notice it memorising.

The flat line has a telltale value. On a yes-or-no task, a model that answers 50/50 every single time scores a loss of about 0.693 (written ln 2 in maths). If the curve sits on that number from step one, the model is not learning slowly. It is not learning at all.

A common misconception

"It is flat, so it just needs more time." In the course's experiment, a deep network with nothing fixed sat at exactly 0.6931 after 800 steps, and was still at exactly 0.6931 after 5,000. Slow training creeps downward. A dead network does not move.

Why it matters

When a run goes wrong, the tempting move is to change four things at once and hope. You might fix it, but you will never find out which change mattered. The loss curve gives you a better starting point: one shape, one likely cause, one thing to check. Change that one thing, run again, and read the new curve. That habit turns guesswork into a method.

Learn it step by step in Chapter 6 of our free course AI From Scratch: Getting It to Train, and Getting It to Generalise.