рд╕рд╛рдордЧреНрд░реА рдкрд░ рдЬрд╛рдПрдБ

рдпрд╣ рд╡реАрдбрд┐рдпреЛ рдФрд░ рдЗрд╕рдХрд╛ рдЯреЗрдХреНрд╕реНрдЯ рдЕрдВрдЧреНрд░реЗрдЬрд╝реА рдореЗрдВ рд╣реИред

Will it fit? When the AI fits your GPU but the conversation doesn't

AI Concepts #191

Fitting an AI model on a graphics card is only half the question. Learn the simple memory sum, and why a 13B model on 8 GB leaves room for barely one prompt.

рд╡рд┐рд╕реНрддрд╛рд░ рд╕реЗ

What "will it fit?" really asks

When people ask whether an AI model will run on their graphics card (GPU), they usually compare two numbers: the size of the model and the memory on the card. That leaves out most of the story. A running model needs memory for three things:

  1. The weights, the model's learned numbers. Their size is the number of parameters times the bytes each one takes. A 7-billion-parameter model (7B) stored at 16 bits per number takes about 13 GB; rounded to 4 bits, about 3.4 GB.
  2. The KV cache, the notes the model keeps about every token it has already read, so it does not have to reread them. A token is a small piece of text. This part grows with every token of every conversation open at the same time.
  3. Runtime overhead, the working space the software itself needs, roughly 1.5 GB.

What the video shows

The video adds it up: an AI needs memory for itself plus notes for every word of every open chat. That is how a model can fit on a graphics card and still leave room for barely one short prompt.

An everyday example

Imagine packing a carry-on bag for a week away. You squeeze in a bulky winter coat and the zip just closes, so technically the coat fits. But there is no space left for the clothes you actually need on the trip. The coat is the model. The clothes are the conversation. And if you are packing for several people, each of them needs their own clothes.

The numbers

The course runs the sum for common card sizes:

| Card | Model | Weights | Tokens of context left | |---|---|---|---| | 8 GB | 7B at 4 bits | 3.4 GB | 25,710 | | 8 GB | 13B at 4 bits | 6.2 GB | 337 | | 16 GB | 7B at 8 bits | 6.5 GB | 65,378 | | 16 GB | 13B at 8 bits | 12.1 GB | 3,136 |

Look at the 13B model on the 8 GB card. The weights fit, so people would say it runs on an 8 GB card. Yet 337 tokens is not a conversation. It is barely a prompt. The overhead can also surprise you: a 7B model at 8 bits weighs 6.5 GB, less than 8 GB, yet the course lists it as not fitting: add the software's 1.5 GB and nothing is left for the conversation.

How a model's design changes the answer

The two 16 GB rows show something no headline mentions. The 13B model gets about twenty times less context than the 7B one, although its weights are only 5.6 GB bigger. The reason is how each model takes notes. In this 13B model, each of its 40 attention heads (parallel readers inside the model) keeps its own set of notes, which costs 800 KB per token. The 7B model uses grouped-query attention, where heads share notes in groups, so it needs only 128 KB per token.

The right question

"Does it fit?" is the wrong question. Ask instead: "With how much context, and for how many users at once?" The notes are shared out across every open conversation, so the 25,710 tokens in the first row could be one long chat or, for example, ten short ones.

Why it matters

A model that loads is not the same as a model you can use. The weights are a fixed cost. The notes grow with every token and every user, and they decide how long and how many conversations the card can actually hold. Before buying a card or picking a model, put the notes into the sum.

Learn it step by step in Chapter 13 of our free course AI From Scratch: Making Inference Cheap: KV Cache, Batching and Quantization.

рдЗрди рдкрд░ рднреА

рдЗрд╕ рдЕрдзреНрдпрд╛рдп рд╕реЗ рдФрд░

рд╣рдорд╛рд░реЗ рдореБрдлреНрдд рдХреЛрд░реНрд╕ рдХреЗ рдЕрдзреНрдпрд╛рдп 13 рд╕реЗ AI рд╕рд╣рд╛рдпрддрд╛ рдХреЗ рд╕рд╛рде рд▓рд┐рдЦрд╛ рдЧрдпрд╛ рдЯреЗрдХреНрд╕реНрдЯред