The LogSumExp trick: one subtraction that stops AI math overflowing

What the LogSumExp trick is

A lot of AI math needs the same small recipe: take a list of scores, raise the number e (about 2.718) to the power of each score, add the results up, and take the logarithm of the total. The name lists the three steps from the outside in: log, sum, exp, where exp is short for exponential.

The trouble is the exponential. It grows absurdly fast: every extra 1 in a score multiplies the result by about 2.7. Once any score goes past roughly 709, the result is bigger than the largest value a standard 64-bit computer number can hold, and the computer hands back infinity.

The trick is one subtraction. Find the largest score, subtract it from every score before exponentiating, and add it back at the end. The answer is mathematically identical, not an approximation, and the biggest term becomes e to the power 0, which is exactly 1. Nothing is left that can overflow.

What the video shows

The video shows how some AI math makes numbers explode past what a computer can hold, and how subtracting the largest value first gives exactly the same answer without the crash.

An everyday example

Imagine adding up amounts in a currency with so many zeros that your calculator runs out of screen. The usual fix is to count in millions: 3 million plus 5 million plus 2 million is just 3 + 5 + 2, with "million" put back at the end. You did not change the answer. You changed the unit, so the numbers stay small while you work. LogSumExp does the same: it measures every term in units of the biggest one, and because of the logarithm, putting that unit back at the end is just adding the largest score.

How it works, with real numbers

The course uses three scores: 1,000, 1,001 and 1,002.

• Written directly: e to the power 1,000 is already far too big, so the result is infinity.
• With the trick: subtract 1,002 from each score to get −2, −1 and 0. Now e to the power −2 is about 0.135, e to the power −1 is about 0.368, and e to the power 0 is 1. They add up to about 1.503, whose logarithm is about 0.408. Add the 1,002 back and the answer is 1,002.408.

The identity holds whatever number you subtract; choosing the largest is what keeps every term at 1 or below.

Where you meet it

The course warns that you will need log-sum-exp constantly. It brings the trick in right after a product of many probabilities has overflowed, as the way to keep calculations with probabilities finite. In Chapter 4 it turns up inside softmax, the step that turns a model's raw scores into probabilities that add up to 100%. There, subtracting the largest score is not a speed-up. It is the only reason softmax gives back a number at all.

A common misconception

It is easy to assume a trick like this trades accuracy for safety. It does not. Subtracting a number and adding it back is exact algebra, so the answer with the trick is the true answer, while the direct version returns no usable answer at all.

Why it matters

Overflow often does not stop your program. In the course's example, the direct formula just returns infinity and the code carries on, so every result that depends on it becomes meaningless. A one-line subtraction prevents that without changing the true answer. It is also a good example of a wider habit: when a formula is correct on paper but fails on a computer, rewrite it into an equivalent form the computer can handle.

Learn it step by step in Chapter 2 of our free course AI From Scratch: Where a Loss Function Comes From: Likelihood, Not Convention.