Target leakage: when an AI model secretly sees the answer

What target leakage is

Target leakage happens when one of the inputs you give an AI model already contains the answer it is supposed to predict. The model is not getting smarter. It is peeking. Because the clue is sitting right there in the data, the scores go up, nothing crashes, and everything looks like a success. Then the model meets the real world, where that clue does not exist yet, and it falls apart.

What the video shows

A model sorts factory parts into "good" and "defective" using two honest measurements: width and weight. It catches only about 11% of the defective parts. Then someone adds one extra column: how many seconds each part spent at the inspection station. Suddenly the model catches 77% of the defects and its overall score more than quadruples.

The catch: a part only lingers at the inspection station because an inspector has already pulled it off the belt as suspect. Good parts pass through in about two seconds; rejected parts sit there for about fifteen. That column is not a measurement of the part. It is a record of the decision the model was supposed to make.

An everyday example

Imagine predicting which students will fail a course, and one of your inputs is "number of resit exams taken". Your model will look brilliant, because resits only happen after someone fails. On the first day of term, when you actually need the prediction, every student has zero resits and the model knows nothing.

The same trap shows up everywhere: a fraud model that uses "account frozen", a churn model that uses "cancellation reason", a medical model that uses "treatment prescribed". Each of those is written down after the outcome is already known.

How to catch it

Ask one question about every input: would this value exist at the moment the prediction is needed? If the honest answer is "only afterwards", remove it.

Two quick checks help:

• Train a model on each input alone. If a single column predicts the answer almost perfectly on its own, treat it as a suspect, not a discovery.
• Do not expect overfitting to warn you. Overfitting shows up as a big gap between training and test scores. A leak does not: the leaked clue is present in both, so both scores look equally great.

Why it matters

Leakage is one of the most common serious mistakes in applied machine learning because it rewards you instead of failing loudly. A number that looks too good deserves a second look, especially when a single new input caused the jump.

Learn it step by step, with the full factory experiment, in Chapter 4 of our free course AI From Scratch: Classification, Cross-Entropy, and How Not to Fool Yourself.