Decision threshold: why a less accurate model can be far more useful

What a decision threshold is

Many AI models do not answer yes or no. They give a probability, such as "there is a 12% chance this part is defective". Something still has to turn that into an action: flag the part or let it pass. That something is the decision threshold, a cut-off line. Above the line, the answer is yes. Below it, the answer is no.

The usual default is 0.5, meaning "flag it if the chance is over 50%". But the threshold is not part of the model. It is a separate choice made after training, and it should be made by whoever understands what each kind of mistake costs.

What the video shows

The video moves that line on a model that looks for defective parts. At the default setting, the model catches 8 of 74 defects. Lower the line, and it catches 71 of 74, even though its accuracy (the share of all parts it labels correctly) goes down. The less accurate version is the far more useful one.

An everyday example

Imagine the smoke alarm in your kitchen. Set it very sensitive and it goes off every time you make toast. Set it too loose and it stays quiet while something is really burning. The sensor inside is the same in both cases. Only the setting changed, and choosing that setting is your call: how many toast alarms will you put up with to be sure you hear about a real fire?

How it works

Here is what the course measured as the line moves down, on a test batch of 4,000 parts with 74 defective ones:

| Threshold | Defects caught (of 74) | False alarms | Accuracy | |---|---|---|---| | 0.5 | 8 | 2 | 98.3% | | 0.2 | 27 | 28 | 98.1% | | 0.05 | 54 | 236 | 93.6% | | 0.005 | 71 | 1,360 | 65.9% |

Look only at the last column and accuracy seems to say: keep the default. Look at the second column and the default is clearly the worst choice for a factory that cannot afford to ship bad parts. Every step that makes this model more useful makes its headline accuracy worse. A team chasing that one number would ship the version that finds almost nothing.

There is a price, of course. Recall (the share of real defects caught) climbs from about 0.11 to 0.96, while precision (the share of alarms that turn out to be real) falls from 0.80 to 0.05. If you want the best balance between the two, measured by the F1 score, it sits near a threshold of 0.2, where F1 reaches 0.42.

A common misconception

When a model misses rare cases, a common first reflex is to retrain it and tell it that the rare cases count more. The course tried exactly that, giving defective parts 10 and then 60 times more weight. Precision and recall moved a lot, but the model had mostly just slid along the same trade-off, as if someone had moved its threshold. That shift can be exactly what you want, but the model learned nothing new about which parts are defective. If a model cannot tell good parts from bad ones in the first place, no weighting and no threshold will fix it.

Why it matters

The threshold is where the model meets the business. Leaving it at 0.5 is itself a decision, even when nobody notices making it. Ask what a false alarm costs, ask what a miss costs, and set the line on purpose.

Learn it step by step in Chapter 4 of our free course AI From Scratch: Classification, Cross-Entropy, and How Not to Fool Yourself.