Quadratic attention: why twice the text means four times the work

What quadratic attention means

Attention is the step inside a language model where each word looks at the other words to work out how they fit together. To do it, the model scores every word against every other word and keeps those scores in a grid. Strictly, models read tokens, small pieces of text that are usually a word or part of one. A text of 1,000 tokens needs a grid of 1,000 rows by 1,000 columns. That is what quadratic means here: the grid grows with the length multiplied by itself. Twice the text, four times the grid.

What the video shows

The short explains that attention compares every word with every other word, so doubling the text roughly quadruples the work. Then it gives the course's own measurement: a single attention layer, not a whole model, ran out of memory on a graphics card with 8 GB at 16,384 tokens. A graphics card (GPU) is the chip that does the heavy maths in AI.

An everyday example

Imagine a teacher filling in a chart that rates how well each student works with every other student. A class of 30 needs 30 × 30 = 900 boxes. A class of 60 needs 3,600. The class doubled; the paperwork quadrupled. Attention fills in that kind of chart for every text it reads, and a long prompt is a very big class.

How the cost grows

The course measured one attention layer on a laptop graphics card, doubling the length each time. Here is the size of the score grid:

| Tokens | Size of the score grid | |---|---| | 1,024 | 32 MB | | 4,096 | 512 MB | | 8,192 | 2 GB | | 16,384 | 8 GB: out of memory |

Every doubling makes the grid four times larger. Measured time and memory both grew close to the length raised to the power of 1.9, which is close to "square the length". At 16,384 tokens the grid alone would take 8 GB: 8 attention heads (parallel copies that each look for different links) times 16,384 times 16,384 numbers, at 4 bytes per number.

Two details keep the story honest:

• Only part of the layer is quadratic. Just two grids grow this way: the raw scores and the weights made from them. The parts that process each token on its own grow roughly in step with the length. In the same test, eight times more text made those parts about ten times slower, not sixty-four.
• Short texts hide the problem. Up to about 512 tokens, those other parts still took most of the time. By 1,024 tokens, the grid had taken over.

Why it matters

This one fact is why a chatbot's context window, the most text it can take in at once, has a limit at all, and why longer inputs cost more. It is also why long prompts become expensive for AI agents, programs that work through a task in many rounds and handle that prompt again and again.

The course adds a fair warning: models also tend to make worse use of very long inputs, but that is a separate finding, not something this formula explains. Tricks like FlashAttention go after the memory side of the problem, which is a story of its own.

Learn it step by step in Chapter 9 of our free course AI From Scratch: Attention and the Transformer Block, Derived From an Average.