Siirry sisältöön

Tämä video ja sen teksti ovat englanniksi.

Concurrency gate: letting fewer requests in made everything faster

AI Concepts #198

Twenty requests rushing a busy AI service got 54 refusals. Letting three through at a time got none and finished eight times faster. Here is why it works.

Tarkemmin

What a concurrency gate is

A concurrency gate is a simple counter that lives in your own app, on your side of the connection. It never lets more than a set number of requests, call it N, be waiting on the AI service at the same moment. When one answer comes back, the next request in line goes out.

Two words worth unpacking. Concurrency just means how many things are happening at once. A request is in flight when it has been sent and its answer has not arrived yet. The gate caps the number in flight. The AI provider does not even need to know the gate exists.

What the video shows

Twenty requests rush a busy AI service at the same time. The service cannot take them all, so it keeps turning them away with a "too many requests" refusal: 54 refusals in total. Then the same twenty pass a gate that lets only three in at a time. This time there are no refusals at all, and everything finishes about eight times faster.

Those numbers come from a measurement in the course. Without the gate, refused requests waited and tried again, so getting twenty answers took 74 requests and about 7.1 seconds. With a gate of 3, it took exactly 20 requests for 20 answers and 883 milliseconds. Less work, and a far shorter wait.

An everyday example

Imagine twenty friends arriving at a small shop with three cashiers. If they all crowd the counter, most are told to wait outside and try again. They step out, wait, and crowd back in, often at the same moment again. Lots of walking, very little buying.

Now imagine they agree at the door: three go in, and each time one comes out, the next one goes in. Nobody is turned away, the cashiers are never idle, and the whole group is done sooner. The rule was set by the friends, not by the shop. That is the gate.

How it works

  • You pick N, the most requests you allow in flight at once. In the course experiment, N was 3. Set it too high and the refusals come back; set it too low and you wait longer than you need to.
  • Before sending, each request takes a slot. If all N slots are busy, it waits in a queue inside your app.
  • When an answer arrives, its slot is freed and the next waiting request goes out straight away.

Waiting in your own queue costs almost nothing. Being refused costs a full trip to the server and back, then a pause before the next try, and then possibly another refusal on top.

Why it matters

Retry rules, like waiting longer after each failure and adding a bit of randomness (the backoff and jitter ideas from this same chapter), are how an app recovers after the service says no. They are still worth having for surprises. But every refusal is wasted time, and when many requests fail together they tend to come back together and fail again.

A gate works on the cause instead of the symptom. It stops your app from asking more than the service can handle in the first place. The short version: a retry is how you apologise to the server, and a gate means you rarely have to.

Learn it step by step, with the full retry experiments, in Chapter 14 of our free course AI From Scratch: Your First Production LLM Call: Streaming, Retries, Timeouts.

Myös palvelussa

Lisää tästä luvusta

Teksti on kirjoitettu AI-avusteisesti ilmaisen kurssimme luvun 14 pohjalta.