Перейти к содержимому

Это видео и текст к нему на английском.

Rate limits: when an AI service says wait, wait at least that long

AI Concepts #199

AI services cap requests and text per minute. Learn what a 429 refusal means, why the wait it asks for is a minimum, and why long prompts hit limits early.

Подробнее

What rate limits are

A rate limit is a cap an AI service puts on how much you can ask of it in a set amount of time, usually per minute. Go over it and the service stops answering for a moment and sends back a refusal instead. That refusal carries the number 429, the standard web code for "too many requests".

The part that is easy to miss is that there are usually two caps running at the same time:

  • Requests per minute: how many separate questions you send.
  • Tokens per minute: how much text those questions contain. A token is a small piece of text, roughly a word or part of a word.

You can be far below the first cap and still hit the second, simply because your prompts are long.

What the video shows

The video makes three quick points. AI services cap both how many requests and how much text you can send each minute. If you go over, they refuse and tell you how long to wait. And that wait should be treated as a minimum, never shortened.

An everyday example

Imagine a post office counter with two rules for each customer: at most ten parcels a day, and at most twenty kilos in total. Turn up with two very heavy boxes and you are turned away, even though two is far below ten. That is a long prompt hitting the text cap while the request count looks harmless.

Now imagine the clerk says "come back in ten minutes". Only the clerk knows when the counter reopens. If your own habit is to come back after two minutes, you will simply be turned away again.

How to respect the wait

When the service refuses, the reply usually includes a small label called Retry-After. It is a header, which means a piece of extra information attached to a web reply, and it says how long to wait. Three rules follow from it:

  1. Treat it as a floor. The service is the only one that knows when its counting window starts over, so never wait less than it asks.
  2. Wait the longer of two times: Retry-After or your own retry delay. The header tells you when the limit forgives you, not when the service will actually have room for you.
  3. Read both formats. Retry-After can be a number of seconds or a calendar date and time. Code that only expects one will misread the other.

In the course's own test, one unlucky app wanted to retry after less than a second, four times in a row. Each time the header asked for a full second, and each time the header won. That app got its answer on the sixth attempt, just under seven seconds after it first tried.

Why it matters

Ignoring the wait turns one refusal into a string of them, and every early attempt is wasted effort for you and for the service. There is also a diagnosis step worth doing. The 429 and its wait look the same whether you sent too many requests or too much text, but the cure is different. For the request cap, you need fewer calls per minute. For the text cap, you need fewer tokens per minute, which can mean shorter prompts even with the same number of calls. Work out which cap you hit before you change anything.

The neighbouring ideas in this chapter, backoff and the concurrency gate, decide how you wait and how much you send. Rate limits are the rule they all have to respect.

Learn it step by step in Chapter 14 of our free course AI From Scratch: Your First Production LLM Call: Streaming, Retries, Timeouts.

Также на

Ещё из этой главы

Текст написан с помощью AI по главе 14 нашего бесплатного курса.