Recurrent networks: how AI read one word at a time before attention

What a recurrent network is

A recurrent network, often shortened to RNN, is an older way for an AI to read text. It takes one token at a time (a token is a word or a piece of a word) and keeps a single running note: a fixed-size list of numbers called its state. After every token it rewrites that note, blending what the note already said with the new token. When the sentence ends, the note is supposed to hold the meaning of everything it read.

For about twenty years, from 1997 to roughly 2017, this was the standard answer to a real problem. Sentences come in any length, but the rest of a model needs one summary of a fixed size. A note that stays the same size however much you feed it fits that job perfectly.

What the video shows

The video shows an older language AI reading word by word and squeezing everything into one note that is rewritten at every step. As the sentence goes on, the early words fade from the note. And because each step has to wait for the one before it, nothing can run in parallel. That is why attention, the approach behind today's chatbots, took over.

An everyday example

Imagine reading a long novel with one strange rule: you may keep a single sticky note, and after every page you must erase it and write a new one, based only on the old note and the page you just read. By page 300, a detail from page 5 is still there only if you carried it forward through almost three hundred rewrites. And nobody can help you: a friend cannot start working on page 150 until your note for page 149 exists.

That is a recurrent network in one picture: one small note, rewritten again and again, strictly in order.

Three ways it broke

The course lists three failures, and the design that replaced it answers each one.

• Early words fade. A model learns by sending a correction signal backwards through every step it took. That signal is called the gradient, and it tells each part of the model how to adjust. Across many steps it tends to shrink to almost nothing or to blow up. The LSTM, a 1997 redesign with a more carefully guarded note, stretched the usable range from tens of steps to hundreds. But the first words still had to survive every single rewrite in between.
• Everything goes through one note. In translation systems, one network read the whole source sentence and handed over only its final note. In 2014, Bahdanau, Cho and Bengio removed that bottleneck: the translator could look back at every note written along the way, one per word, and take a weighted mix of them, choosing the weights itself. That was attention, three years before the transformer.
• No teamwork. Each step needs the previous note first. A graphics chip (GPU) has thousands of small cores built to do many calculations at the same moment, and a strictly one-after-another process leaves almost all of them waiting.

Why it matters

The transformer did not win because it is obviously smarter. It won because its expensive work is matrix multiplication: big grids of multiply-and-add that a GPU can run all at once. A design that fits the hardware beats a design that keeps the hardware idle.

This also explains a choice the course makes on purpose: it never builds a recurrent network. It names the approach, explains the three failures, and moves straight to attention, which keeps the one great idea from 2014 (look back at every word and choose how much each one counts) and deletes the step-by-step rewriting.

Learn it step by step in Chapter 9 of our free course AI From Scratch: Attention and the Transformer Block, Derived From an Average.