Pretraining an LLM: Data, Compute, Scaling Laws and Cost

Nothing in the objective changes. The model still predicts the next token, the loss is still the cross-entropy of Chapter 4 applied to the factorisation of Chapter 8, the optimiser is still the AdamW of Chapter 6. Pretraining is not a new algorithm; it is the same algorithm run on a corpus large enough that the run has to be budgeted. Two things make that possible: the labels are free, since the target for position $t$ is the token at $t+1$ and is already in the text; and Chapter 6's last section removed the objection, because a model with far more parameters than the classical rules allow does not fall apart, it improves. What comes out is a base model — something that continues text rather than answering.

Before any of this can be budgeted it has to be counted, and the field counts it with one formula:

where $N$ is the parameter count, $D$ the training tokens and $C$ the total floating-point operations. Kaplan et al. derive it in two steps.¹ Forward: 2 FLOPs per parameter per token, since every parameter in a matrix multiply is used once per token, in one multiply and one add. Backward: twice the forward, since the backward pass of Chapter 5 computes two gradients at each layer — with respect to the layer's inputs, so the signal keeps travelling, and with respect to its weights — each a matrix multiply the size of the forward one, so $4N$.

That is the whole derivation, and it is worth checking rather than believing. PyTorch ships a real FLOP counter, torch.utils.flop_counter.FlopCounterMode, which intercepts every operation a model dispatches and totals the actual work. Run it across four orders of magnitude, the largest on the meta device, which allocates shapes and no memory:

from torch.utils.flop_counter import FlopCounterMode

counter = FlopCounterMode(display=False)
with counter:                       
    loss = model(x, targets)[1]     
    loss.backward()                 
measured = counter.get_total_flops()
print(measured / (6 * n_params * n_tokens))

The forward+backward over forward ratio is 3.000, exactly, at every scale: not an approximation that happens to be good, but the arithmetic identity above returned as a round number by a counter that knows nothing about the derivation.

The measured total then sits between 3 % and 17 % above $6ND$, once $N$ counts the embedding matrices — and that clause matters, because the two founding papers count $N$ differently. Kaplan excludes "all vocabulary and positional embeddings" because doing so "produces significantly cleaner scaling laws" (§1.3); Chinchilla's Appendix F says "we also count embeddings matrices in the total parameter count".² For a wide vocabulary and a narrow hidden dimension the two differ by a factor of nine, as the first row shows.

The residual gap is what $6ND$ deliberately omits: the attention scores. Kaplan's Eq. (2.2) writes the forward cost as $2N + 2\,n_{\text{layer}} n_{\text{ctx}} d_{\text{model}}$ and drops the second term because $d_{\text{model}} \gg n_{\text{ctx}}/12$ — safe in 2020, less safe now, and the reason the ratio drifts upward as $T/d$ grows — which is why two rows here share a $T$ of 1,024 and the ratio falls, from 1.145 to 1.100, when $d$ goes from 768 to 1,600. It is the $O(T^2)$ cost Chapter 9 introduced and Chapter 16 turns into a price.

Compute decides how long a run takes; memory decides whether it can start. Train with plain fp32 AdamW and every parameter carries four numbers: the weight, its gradient, and Adam's running mean $m$ and variance $v$ — the two averages built by hand in Chapter 6. Four numbers at four bytes each is 16 bytes per parameter, before a single activation. Measured on an 8 GB laptop GPU, taking the resident allocation at the point in the step where no graph is alive:

Prediction and measurement agree to within 3 %. The surprise is the last column: the activations dwarf the model. The same 5.8-million-parameter model that needs 89 MB of persistent state needs 4,681 MB of activations at a batch of 128 — fifty-two times the model — and much of that is not the transformer at all. It is the logits, one vector of vocabulary size per token at four bytes an entry: 512 MB in the last row, 393 MB in the first. The vocabulary size was chosen in Chapter 7, and it is still deciding what fits on the card.

Which term dominates depends on the shape of the run, which is why Micikevicius et al. say memory "is dominated by activations"³ while ZeRO says a 1.5-billion-parameter model needs "at least 24 GB" of model states alone.⁴ ZeRO reaches the same 16 bytes by another route — $2\Psi$ for fp16 weights, $2\Psi$ for fp16 gradients, $4\Psi$ each for the fp32 master weights and Adam's two moments — which for 70 billion parameters is 1.12 terabytes, fourteen 80 GB GPUs' worth before a single activation.

None of that fits on one device at frontier scale, so the run is split four ways at once. Data parallelism puts a copy of the model on every GPU and averages the gradients — the default, and the one ZeRO improves by refusing to keep redundant copies of the optimiser state. Tensor parallelism splits individual matrices across devices. Pipeline parallelism gives each device a contiguous group of layers. Context parallelism splits the sequence itself, necessary only once $T$ is long enough for the attention term to dominate. Llama 3's Table 4 lists all four at once: tensor 8, context up to 16, pipeline 16, data up to 128, across 16,384 H100 GPUs.⁵ That is all this course will say about it; distributed training engineering is a semester of its own, and Stanford's CS336 is that semester, lectures 5 to 8, with the code.⁶ What survives the delegation is a single number, model FLOPs utilisation — the fraction of a GPU's peak arithmetic a real run achieves — which is what turns the tidy $6ND$ into wall-clock time and therefore into money.

In January 2020, Kaplan et al. trained a grid of transformers and found the test loss follows a power law in each of the three resources over more than six orders of magnitude.¹ Their §1.2 gives three fitted laws:

with companions $\alpha_D \approx 0.095$ for data and $\alpha_C^{\min} \approx 0.050$ for optimally allocated compute. The constants are not universal, and the paper says so: "the precise numerical values of $N_c$, $C_c^{\min}$ and $D_c$ depend on the vocabulary size and tokenization and hence do not have a fundamental meaning."

The exponents are tiny: ten times the parameters buys a factor $10^{0.076} \approx 1.19$ off the remaining loss. That sounds like nothing, and it is the most important fact here — the returns are terrible and they never stop. A power law with a small exponent promises that the next order of magnitude will help, less than the last one did, forever. Buying compute stops being a gamble and becomes a purchase with a published exchange rate, which is exactly the argument that unlocked the capital.

Then came the prescription, and this is where the paper was wrong in a way that cost the industry a great deal of money. Kaplan's Table 6 gives $N_{\text{opt}} \propto C^{0.73}$ and $D_{\text{opt}} \propto C^{0.27}$: ten times the compute means a model 5.4 times bigger fed only 1.9 times as much text. The abstract is explicit — "optimally compute-efficient training involves training very large models on a relatively modest amount of data and stopping significantly before convergence." The field did exactly that: GPT-3 is 175 billion parameters on 300 billion tokens,⁷ Gopher 280 billion on 300 billion, Megatron-Turing NLG 530 billion on 270 billion.² Half a token to two tokens per parameter, across the board.

In March 2022, Hoffmann et al. trained over 400 models from 70 million to 16 billion parameters and reached the opposite conclusion by three independent routes.² Their Table 2 reports the exponent $a$ in $N_{\text{opt}} \propto C^{a}$ as 0.50, 0.49 and 0.46, against Kaplan's 0.73. In plain terms: model size and training data should grow in equal proportion.

Their second approach is the one reproduced by the sweep at the top of this chapter, at a millionth of the scale: fix a budget, train many sizes at exactly that budget, plot final loss against model size.

Held-out loss in nats per token, tokens per parameter in brackets, bold for the best model at each budget; a dash is a point not run, because the budget demanded more text than the corpus holds or the size fell outside those swept there.

Read down a column: the loss falls, bottoms out and climbs again. A model can be too big for its budget exactly as easily as too small — at $10^{14}$ the penalty for choosing 665,280 parameters over 295,808 is 0.08 nats, which on the envelope fitted above is the loss a correctly sized model reaches with 18 % less compute. Picking the wrong shape throws away a fifth of the budget. That is Chinchilla's Figure 3 in an afternoon on one GPU instead of with four hundred models.

Now read across. At $10^{13}$ the best model is the smallest one swept; at $10^{14}$ it is 295,808 parameters, bracketed on both sides. The optimum moves right as the budget grows, which is the whole content of the correction. Fit the paper's third approach — the surface $L(N,D) = E + A/N^{\alpha} + B/D^{\beta}$ over every run — and minimise subject to $C = 6ND$:

L(N, D) = 24.7 / N^0.195 + 46.8 / D^0.169       (E fits to ~0; see below)
implied   N_opt ∝ C^0.464
  compare   Chinchilla 0.46-0.50 · Besiroglu 0.513 · Kaplan 0.73

0.46, from a laptop, against Kaplan's 0.73. Agreement to three digits from a three-budget fit is luck; agreement to the first is not. The exponent travels — the constant does not, since the token-to-parameter ratio at these optima is 170 to 540, not 20. Three reasons, all instructive. $E$ fits to zero because at a loss above 4 nats the run is nowhere near the entropy floor that dominates Chinchilla's fit. Batch size and learning rate were fixed rather than tuned per point, which handicaps whichever runs get fewest steps — and those are the large models: at $10^{13}$ FLOPs a 1.28-million-parameter model gets 159 optimiser steps in total, far under the few thousand Kaplan's $S_{\min}$ term says any model needs. A scaling law is fitted inside a regime, and this one sits six orders of magnitude below Chinchilla's.

Hence the paper's abstract: "current large language models are significantly undertrained". Chinchilla is the demonstration — 70 billion parameters on 1.4 trillion tokens, the same total compute as Gopher's 280 billion on 300 billion, beating it on 51 of 57 MMLU tasks, 67.5 % against 60 %.² Four times smaller, four and a half times more text, same money, better model.

Two caveats on that famous ratio. "Twenty tokens per parameter" is not a sentence in the paper, which says only that "for every doubling of model size the number of training tokens should also be doubled"; the 20 is an inference from Table 3 and from Chinchilla's own 70 B on 1.4 T. And its precision is worse than published: Besiroglu et al. refitted from a digitisation of Figure 4, found the original parameters "fit the reconstructed data poorly" with intervals "implausibly tight given the number of data points", and put the honest range at "between 4 and 40" tokens per parameter.⁸

One detail of Chinchilla's method pays off a promise Chapter 1 made about learning-rate schedules. The cosine schedule has to be matched to the token budget. A model that will see 10 million tokens must decay its learning rate to zero at 10 million tokens; give it a schedule sized for 100 million, stop it early, and you are reading a loss mid-descent at a rate far too high. Chinchilla trains each model at four cycle lengths to control for exactly this; the sweep above sets its schedule from the budget for the same reason.

They are the most useful empirical result in the field and they are routinely oversold. Four limits.

They predict loss, not capability. The left-hand side is cross-entropy on held-out text. Nothing in these papers licenses a claim about whether a model will write correct SQL, refuse a harmful request or use a tool. This is Chapter 5's lesson again: a prediction of the loss is not a prediction of the behaviour you are paying for.

They are fitted, not derived. No theory produces $\alpha_N = 0.076$. The constants move with the tokenizer — which is why a perplexity comparison across two tokenizers is meaningless, as Chapter 8 explained — and with the data mixture, architecture and optimiser. Every published law is a law of the setup that produced it, which is why Meta refitted its own before Llama 3.⁵

They assume a fresh token for every step, which quietly assumes an infinite corpus. Muennighoff et al. measured what happens when it runs out: up to four epochs of repeated data cost almost nothing — an 8.7-billion-parameter model on 44 billion unique tokens seen four times finished "only 0.5 % higher validation loss" than the same model on 178 billion unique ones — while past about sixteen epochs additional compute buys nothing.⁹

And nobody trains compute-optimal any more. Chinchilla minimises the cost of training; a deployed model then pays roughly $2N$ FLOPs per generated token, forever. LLaMA 1 said it plainly: "given a target level of performance, the preferred model is not the fastest to train but the fastest at inference".¹⁰ Sardana et al. formalised it by minimising $6ND_{\text{train}} + 2ND_{\text{inference}}$ instead, and found anyone expecting a billion requests should train "smaller and longer than Chinchilla-optimal".¹¹ Llama 3's §9.1 agrees: its small models train "far beyond the point of compute optimal training, effectively trading training compute for inference efficiency".⁵ The ratio is not obsolete; it answers a question that is no longer the one being asked.

Loss falls smoothly. Benchmark scores sometimes do not. Wei et al. collected cases where a task sits at chance across orders of magnitude of training compute and then jumps — three-digit arithmetic appearing in GPT-3 at about $2 \times 10^{22}$ FLOPs, MMLU rising above guessing between $3$ and $5 \times 10^{23}$ — and named the pattern: "an ability is emergent if it is not present in smaller models but is present in larger models".¹² If that is a real property, extrapolating from cheap experiments is unsafe, because the capability you are buying may not exist at any scale you can afford to test.

Schaeffer, Miranda and Koyejo argued that most of it is an artefact of measurement, and the mechanism is arithmetic.¹³ Per-token loss falls smoothly, so the probability of one token being right, $\exp(-\mathcal{L})$, improves gradually. Score the model with exact string match over an $L$-token answer and you raise that probability to the power $L$ — a smooth curve raised to a large power looks like a cliff. Swap in a metric that counts tokens instead of demanding all of them, on the same outputs, and "the family's performance smoothly, continuously and predictably improves with increasing scale".

Their audit is the number to remember — "of the 39 preferred metrics in BIG-Bench, at most 5 display emergence", with two discontinuous metrics accounting for over 92 % of claimed cases — and so is their caution: "nothing in this paper should be interpreted as claiming that large language models cannot display emergent abilities". A jump in a chart is evidence about the metric until shown otherwise. Chapter 29 is where that becomes your problem, because choosing a hard-cutoff metric is a decision you will make without noticing.

The corpus is the part of a pretraining run with no equation attached, and where most of the consequential decisions live. The raw material is a web crawl: Common Crawl's August 2026 archive holds "2.14 billion web pages or 360 TiB of uncompressed content", one month of it, free to download.¹⁴ Almost none is usable as it stands. The T5 paper says the crawl "largely comprises gibberish or boiler-plate text like menus, error messages, or duplicate text", and the C4 pipeline it introduced is a list of blunt heuristics — keep only lines ending in terminal punctuation, drop pages with fewer than three sentences, drop any page containing a curly brace or a word from a public list of obscenities — turning twenty terabytes of monthly text into about 750 GB.¹⁵

Blunt is the word. Dodge et al. audited what those filters remove and found the obscenity blocklist deletes 42 % of documents in African-American English and 32 % in Hispanic-aligned English, against 6.2 % of White-aligned English, leaving a corpus 97.8 % of the last category.¹⁶ A rule with no opinion about dialect had one.

Then deduplication, which is not housekeeping: Lee et al. found a 61-word sentence repeated 61,036 times in C4, and showed deduplicating cuts the rate at which models "emit memorized text" tenfold, from 1.9 % of generated tokens to 0.19 %.¹⁷ More is not better, though — FineWeb's team deduplicated globally across 96 crawls, got 4 trillion tokens and no measurable gain, then deduplicated each crawl separately, got 20 trillion, and matched the best existing corpus.¹⁸

Then contamination. Llama 3 measured its own and published it: 98 % of AGIEval, 95 % of BIG-Bench Hard and 85 % of HellaSwag overlapping the training set by 8-grams, and for MMLU an overlap so high that "it is impossible to get a good performance gain estimate".⁵ GPT-3's §4 reports a filtering bug that left benchmarks in the data with no way back: "because of cost considerations it was infeasible to retrain the model".⁷

Provenance is the unresolved part. The Pile shipped a 100.96 GiB component called Books3 — 12 % of the corpus and, by the paper's own consent table, books from a private torrent tracker;¹⁹ it was taken offline in August 2023 after a copyright complaint. The legal position as of September 2026 is unsettled, and the three US rulings cited as a trend disagree with one another. Alsup found training on lawfully acquired books "exceedingly transformative" while holding that a library built from pirated copies was not, and Anthropic settled that half for $1.5 billion covering 482,460 works, roughly $3,000 each, approved 20 July 2026.²⁰ Chhabria granted Meta summary judgment while writing that his ruling "does not stand for the proposition that Meta's use of copyrighted materials to train its language models is lawful", only that "these plaintiffs made the wrong arguments".²¹ Bibas, ruling against Ross Intelligence, noted that "only non-generative AI is before me today".²² No US appellate court has ruled on the question.

People do the parts the loss cannot. TIME reported in January 2023 that workers labelling toxic text for OpenAI through the firm Sama took home "between around $1.32 and $2 per hour" reading passages describing child sexual abuse, torture and self-harm, while OpenAI paid Sama $12.50 an hour for the work; Sama disputes both the pay range and the quota.²³ That is the filtering around pretraining rather than pretraining itself — but it is on the same invoice, and it is where a person sits.

The electricity is real and usually misquoted. The most careful published figure is BLOOM's: 1,082,990 GPU-hours, 433 MWh and 24.7 tonnes of CO₂ equivalent for the run, 50.5 counting manufacturing and idle nodes;²⁴ Patterson et al. put GPT-3 at 1,287 MWh and 552 tonnes.²⁵ Two cautions. BLOOM's advantage is its French nuclear grid at 57 g CO₂ per kWh rather than efficiency — it used more energy than OPT-175B. And the field's most-quoted emissions figure, Strubell et al.'s 626,155 lb for a neural architecture search, was later shown to be 88 times too high, having assumed the search ran at full model size when it ran on a proxy.²⁶ LBNL's framing is the defensible one: US data centres used 192 TWh in 2024, 4.7 % of national electricity — a number attached to an industry, not to any one run.²⁷

The through-line is what Bender et al. named documentation debt: "putting ourselves in a situation where the datasets are both undocumented and too large to document post hoc".²⁸ Every fact above exists because somebody looked. For the corpora behind the models most people use, nobody can.

Now the arithmetic everyone wants, from four cited inputs, so that when they go stale it is obvious which to replace.

Read the last two columns together. At $10,000 you get a 5-billion-parameter model on one rented node in a fortnight. At $100,000,000 the arithmetic says 546 billion parameters — and twelve thousand H100s wired together for three months, which is not something you rent with a credit card. Past about $100,000 the binding constraint stops being money and becomes the cluster.

Before trusting a table like that, test it against runs whose real cost is published — llm.c reproduces GPT-2 124M in "~90 minutes" on an 8×A100 node "for about $20", and GPT-2 1.6B in 24 hours on an 8×H100 node for $672.³¹

$672, against what $672 actually bought (llm.c GPT-2 1.6B, one 8xH100 node, 24 h)
  this table predicts:      168 H100-hours   N = 1.41 B params   D = 28.3 B tokens
  what was actually run:    192 H100-hours   N = 1.558 B params  D = 33.6 B tokens

Llama 3 405B, against Meta's own published GPU-hours
  from the paper's 3.8e25 FLOPs at 40 % MFU:   26.67 M H100-hours
  published in Meta's Llama 3.1 model card:    30.84 M H100-hours    ratio 0.86

Both within about 15 %, which is roughly the accuracy this kind of estimate deserves and considerably better than the accuracy it is usually quoted with.

The most repeated figure in this subject is that a GPT-2-class model costing about $43,000 in 2019 can be reproduced today for a few tens of dollars. The modern half is well documented; the historical half is not.

Today. Karpathy's nanochat README: "you can train your own GPT-2 capability LLM ... for only $48 (~2 hours of 8XH100 GPU node) ... On a spot instance, the total cost can be closer to ~$15."³² "GPT-2 capability" here is precise and published — beating GPT-2's CORE score of 0.256525 — on a leaderboard whose best entry as of 14 March 2026 is 1.65 hours. The $48 assumes $3 per GPU-hour, below Lambda's list $3.99; at list it is nearer $64.

In 2019. There is no primary source: OpenAI never published a duration or a cost. The chain runs The Register, February 2019, reporting "256 Google TPU3 cores" with no price and no duration; then Synced, June 2019, noting that hardware cost $256 an hour on Google Cloud and stating explicitly that "OpenAI didn't specify the training duration". $43,008 is $256 an hour times an assumed 168 hours that nobody has ever sourced.

So the honest headline is: a model matching GPT-2's published benchmark score can be trained today for well under $100 on rented hardware, against a 2019 cost that was never published and whose famous estimate rests on an unsourced guess about the duration. The collapse is real and the modern half is reproducible by anyone with a credit card; the ratio is arithmetic on a number that does not exist. That is the state of published training costs generally. The GPT-3 paper contains no dollar amount at all, only $3.14 \times 10^{23}$ FLOPs in Table D.1;⁷ the Llama 3 paper contains none either.⁵ Every training cost you have read is an estimate from a FLOP count, a hardware assumption and a price assumption — always worth asking whose.

What comes out has seen a fixed corpus assembled at a fixed moment, and two properties follow.

The first is the knowledge cutoff. After the collection date the model knows nothing — not "is uncertain", nothing — and it will confabulate fluently rather than say so, because saying so was never a behaviour it was trained on. Llama 3.1's model card gives December 2023;³³ every model has one, and it is a property of the training data, not the deployment. Working around it is a retrieval problem, which is Chapter 19.

The second is that a base model completes rather than answers. Give it "What is the capital of France?" and a plausible continuation is another question, because in the corpus that string most often appears in a list of exercises.

A text completer is not an assistant. It does not follow instructions, because nothing in the corpus told it a request should be obeyed rather than continued. It has no notion of a conversation with two participants. It will happily produce the most probable continuation of a harmful prompt, because probable is the only thing it was ever optimised for.

Turning it into something that answers takes a second stage costing a fraction of a per cent of the first, and consisting almost entirely of showing it examples of the behaviour you want and then comparing pairs of its own outputs. That stage is where instruction following, chat templates, refusals and — this surprises people — the ability to call a tool all come from. Chapter 11 is that stage: supervised fine-tuning, RLHF, DPO and GRPO, and the question of what "aligned" means and who decides.

Also worth reading alongside this chapter: Karpathy's build-nanogpt and its accompanying video, which walk a full GPT-2 reproduction end to end at a pace this chapter cannot; and Stanford CS324, Large Language Models, whose lectures on data and on environmental impact go deeper than the section above into material this course treats once and delegates.