#192Este vídeo e o respetivo texto estão em inglês.
Thundering herd: when retrying apps move in lockstep and jam a server
When many apps fail at once and retry on the same schedule, they keep colliding. One line of randomness took a test from 226 seconds to 2.2. Here is why it works.
Em detalhe
What a thundering herd is
A thundering herd happens when a crowd of apps all hit a limit at the same moment and then retry on exactly the same schedule. They fail together, wait the same amount of time, come back together and fail together again. Nothing in the waiting breaks them apart, so the whole crowd keeps charging at the server as one block.
It usually starts with a traffic spike, which by definition means many requests arriving at once. The retry rule meant to help, waiting longer after each failure (exponential backoff), keeps the crowd in step when everyone follows it from the same starting point: everyone waits 200 milliseconds, everyone retries, everyone fails, everyone waits 400.
What the video shows
The video's point: when many apps retry on the same schedule, they keep crashing into each other. Giving each one a random wait spreads them out. In the course's test, that change took the time needed to serve the whole crowd from about 226 seconds to about 2.2 (the middle result of three runs each way).
An everyday example
Imagine a lift that fits three people and a hundred office workers who all press the button at 9:00. Three get in, and ninety-seven are told to try again in one minute. At 9:01 all ninety-seven come back at once. Three get in, the rest are told two minutes. The crowd never thins out between trips, so every arrival is a scramble. Now imagine each person instead picks a random moment within that minute. They drift in a few at a time, and the lift carries them steadily.
What the test found
The course ran 100 simulated apps against a server that handles three requests at a time, three runs each way:
| Retry style | Time to serve everyone | Most arrivals in one 50 ms window | |---|---|---| | Same schedule for all | 65.6 s, 245.7 s, 225.6 s | 46 to 97 | | Random wait for each | 2.2 s, 2.3 s, 1.8 s | 25 to 32 |
Two things stand out:
- Clumping. Without randomness, up to 97 of the 100 apps landed inside the same 50-millisecond slot, while the server had room for three. With randomness they arrived in groups of around thirty and cleared almost at once, using fewer than half the requests.
- Unpredictability. The synchronised runs did not just take longer; they swung wildly, from about one minute to about four. Tiny timing accidents decided which three apps got in first, and everything else followed from that.
How to spot it
In a live product, a herd has a recognisable signature: a service that is fine, fine, fine, and then suddenly takes four minutes, with no change on your side to explain it. Rare, huge, random slowdowns after a busy moment are a strong hint that your retries are moving in lockstep.
How to fix it
Randomise the wait. Instead of sleeping exactly the scheduled time, each app sleeps a random amount between zero and that time. The course calls this full jitter, and it takes one call to a random-number function. The crowd stops moving as one, and the server sees a steady trickle instead of a stampede.
Why it matters
Traffic spikes are exactly when your app most needs to recover quickly, and exactly when a herd forms. A single line of randomness turns that stampede into an orderly queue, and it makes your system predictable again.
Learn it step by step in Chapter 14 of our free course AI From Scratch: Your First Production LLM Call: Streaming, Retries, Timeouts.
Também em
Mais deste capítulo
#192
#193Streaming: why AI answers arrive word by word, and what it really buys
#194Finish reason: telling a finished AI answer from a cut-off one
#195Error triage: which AI errors deserve a retry and which never do
#196Exponential backoff: why a retry should wait twice as long each time
#198Concurrency gate: letting fewer requests in made everything faster
Texto escrito com assistência de IA a partir do capítulo 14 do nosso curso gratuito.