#192این ویدیو و متن آن به انگلیسی هستند.
Streaming: why AI answers arrive word by word, and what it really buys
Streaming does not make an AI answer finish any sooner. In the course's test the first word appeared 12 times earlier. Here is what that changes for the reader.
با جزئیات
What streaming is
A chat AI writes its answer in small pieces called tokens, which are words or parts of words. Your app has two choices. It can wait until every piece is ready and then show the whole answer at once, or it can show each piece the moment it arrives. The second option is called streaming, and it is why chat answers seem to type themselves on screen.
What the video shows
The video corrects a popular belief: streaming does not make the AI faster. Showing the answer piece by piece does not finish it any sooner. What changes is when the first word appears. In the course's test it arrived 12 times earlier, so people spend that time reading instead of staring at a spinner.
The numbers behind it
The course asked the same question with and without streaming, using a practice server that produced 13 tokens, one every 60 milliseconds (a millisecond is a thousandth of a second).
| Method | First word on screen | Full answer | |---|---|---| | Wait for everything | 791 ms | 791 ms | | Streaming | 65 ms | 793 ms |
Without streaming, both numbers are the same, and that is exactly the problem. For most of a second the user sees nothing, even though the server had the first words ready almost at once. With streaming, the full answer lands 2 milliseconds later, which nobody can notice, while the first word shows up about twelve times sooner.
An everyday example
Imagine following a football match. You can listen to the live radio commentary, or you can wait for a full match report handed to you at the final whistle. The match ends at the same moment either way. With commentary, you knew what was happening the whole time. Without it, you sat through ninety minutes of silence and then got the whole story in one go.
How it works
The server sends the answer as a series of small messages, each carrying the next bit of text, over one connection that stays open. The course uses a standard web method for this called Server-Sent Events. The app on the receiving end needs carefully written code to read that flow, because the network does not respect where one message ends and the next begins:
- One delivery can hold half a message, or two and a half. The reader keeps the leftovers and waits for the rest.
- One letter can be split in two. Accented letters and emoji take several bytes, and those bytes can arrive in different deliveries. Handled carelessly, they turn into garbage symbols at random.
- Each message ends with a blank line, not a single line break, so that is what the reader looks for.
Why it matters
A screen that fills up feels very different from a spinner, even when the total time is identical. Streaming does the same work in the same time but turns waiting into reading. That is the whole benefit, it is a big one, and it is the reason every chat product streams its answers.
Learn it step by step in Chapter 14 of our free course AI From Scratch: Your First Production LLM Call: Streaming, Retries, Timeouts.
همچنین در
بیشتر از این فصل
#192
#194Finish reason: telling a finished AI answer from a cut-off one
#195Error triage: which AI errors deserve a retry and which never do
#196Exponential backoff: why a retry should wait twice as long each time
#197Thundering herd: when retrying apps move in lockstep and jam a server
#198Concurrency gate: letting fewer requests in made everything faster
متن با کمک AI و بر اساس فصل 14 از دوره رایگان ما نوشته شده است.