Към съдържанието

Това видео и текстът му са на английски.

Exponential backoff: why a retry should wait twice as long each time

AI Concepts #196

When an AI service is overloaded, coming straight back makes it worse. Exponential backoff doubles the wait after each failure. Here is how it works, and its catch.

В детайли

What exponential backoff is

Sometimes an AI service is too busy to answer, and your app has to try again later. Exponential backoff is the standard rule for how long "later" should be:

  1. After the first failure, wait a short starting delay.
  2. After each new failure, double the wait.
  3. Stop growing at a maximum, so nobody waits forever.

"Exponential" simply means the wait multiplies instead of adding up. In the course's code the starting delay is 200 milliseconds (a millisecond is a thousandth of a second) and the ceiling is 20 seconds.

What the video shows

The video's message is short: when an AI service is overloaded, apps should wait before retrying and double the wait each time they fail again. Coming straight back only makes the overload worse.

Why not just retry right away?

An overloaded server is already struggling to keep up. Every app that fails and instantly tries again adds one more request to a pile the server cannot handle, so the pile grows instead of shrinking. Backing off gives the server room to recover: each failed app steps away, and steps further away if the problem continues.

An everyday example

Imagine a crowded café where the barista says "come back in a bit". If you walk up to the counter every ten seconds, you add to the crush and slow everyone down, including yourself. A sensible customer comes back in two minutes, then four, then eight, and after a while stops stretching the gap and just checks in every so often. That is exponential backoff with a ceiling.

How it works

Three settings define it:

  • Starting delay: the first wait, 200 milliseconds in the course.
  • Doubling: every new failure doubles the previous wait.
  • Ceiling: the wait never grows past a cap, 20 seconds in the course.

With those settings the waits run 200, 400, 800, 1,600, 3,200, 6,400 and 12,800 milliseconds. The next doubling would pass the cap, so from the eighth wait onwards every pause is 20 seconds.

Backoff only makes sense for problems that time can fix, such as "too busy" or "temporarily broken". If the request itself is wrong, no amount of waiting helps. Sorting errors into those two groups is a concept of its own, error triage.

The catch

Plain backoff has one weak spot: everyone doubles from the same starting point. When a traffic spike makes many apps fail at the same instant, they wait the same time, come back together and collide again. That pile-up has its own name, the thundering herd. The usual cure is small: each app waits a random amount between zero and its computed time, a trick the course calls full jitter.

Why it matters

Retries let an app ride out a busy moment without bothering the user. Done carelessly, they turn a busy moment into a longer one. Doubling the wait, capping it and adding a pinch of randomness is how an app stays persistent without becoming part of the problem.

Learn it step by step in Chapter 14 of our free course AI From Scratch: Your First Production LLM Call: Streaming, Retries, Timeouts.

Също в

Още от тази глава

Текстът е написан с помощ от AI въз основа на глава 14 от безплатния ни курс.