Temperature, Top-p and the Determinism You Do Not Have

Chapter 4 introduced the logit as an unnormalised real-valued score, one per class. Chapter 8 made a language model produce one per vocabulary entry. The softmax turns that vector $\mathbf{z}$ into probabilities:

Temperature enters here — the name is borrowed from statistical physics, where the same parameter controls how sharply a Boltzmann distribution concentrates on its low-energy states¹ — and it divides the logits before the exponential:

That placement is the entire mechanism, and it is worth two lines of algebra to see why it could not be anywhere else. Suppose you tried to apply temperature to the probabilities instead — scale them by $1/T$ and renormalise. You would get

The constant cancels. Scaling probabilities does nothing at all; the distribution comes back unchanged. Temperature only has an effect because it acts on the exponent, where dividing by $T$ before exponentiating is the same as raising each probability to the power $1/T$ — a nonlinear reshaping that changes the ratios between entries rather than their common scale.

From that placement, both limits follow without any further work. As $T \to 0$ the largest logit runs away from the rest and $p$ collapses onto the single highest-scoring token: greedy decoding. As $T$ grows every $z_i/T$ heads towards zero, every exponential heads towards 1, and the distribution flattens towards uniform over the whole vocabulary. At exactly $T = 0$ the formula divides by zero, so every implementation special-cases it to the arithmetic maximum — including the widget below, which switches to argmax at $T \le 0.001$.

One warning, because the collision of names causes real confusion. There is a second, unrelated thing called temperature in machine learning: temperature scaling, a calibration method that fits one value on a validation set so a classifier's confidence matches its accuracy.² Same formula, nothing to do with generation. Papers saying "temperature" often mean that one; this chapter never does.

Here is that distribution, with the arithmetic in front of you. The logits are fixed and plausible, so the numbers in the prose below can be checked against what you see:

The ␣banana number is the whole argument in miniature: raising the temperature cannot give a model an idea it did not have. The logits are already computed, the ranking is already fixed, and temperature preserves it exactly — no amount of heat ever moves a lower-scored token above a higher-scored one. All it does is redistribute mass down the ranking the model itself produced. High temperature does not make a model more inventive; it makes it more likely to emit the tokens it scored as bad.

On a real vocabulary this stops being a curiosity and becomes the reason high-temperature output is unusable. Measured on Qwen/Qwen2.5-0.5B-Instruct, one forward pass, the prompt above, counting how many tokens it takes to accumulate a given share of the probability mass:

Read the bottom row slowly. At $T = 2$, on a question with exactly one correct answer, 32,966 different tokens share the top 90 % of the probability mass. That is not a wider creative space. That is a model that has been told, by arithmetic, to treat a Korean particle and a C++ identifier as live options for the word after A:. The garbage in the opening block is the direct consequence, and it is not a bug in the model or the library — it is what the request asked for.

The useful range is narrow and depends on the task rather than on taste. On a factual question the answer is one token and any heat above about 1.2 injects error for nothing. On an open-ended one there really is more than one good continuation, and some heat buys variety that stays fluent:

"Write a two-sentence story about a lighthouse."

T = 0.0  "The lighthouse stood tall and proud, its beacon illuminating the
          night sky above. A lone sailor, his eyes fixed on the distant
          horizon..."

T = 0.7  "In the quiet, stormy waters of the sea, a lighthouse stood
          sentinel over the horizon, its golden dome casting a warm glow
          on the fog-shrouded streets below..."

T = 1.0  "In the quiet night, a lone lighthouse stood sentinel over the
          sea, its shining beacon a beacon of hope and solace for sailors
          and fishermen across the vast and endless ocean..."

T = 1.3  "In the gentle sunlight, now reflecting upon the opening of Jack's
          lighthouse, Jim Trahan, a small-time individual difficult to
          define in paperwork, wondered about a career where simplicity
          reigns..."

At 1.3 the model has invented a proper name and a sentence that does not parse. The band between "identical every time" and "incoherent" is roughly 0.6 to 1.1 for this model on this task, and the honest advice is that you find it by measuring on your task, not by copying a number from a blog post.

There is an obvious question hiding under all of this: if the model has a probability distribution and one token is most likely, why not always take it? Greedy decoding is free, reproducible and needs no parameters.

Because the result is this:

prompt: "In a shocking finding, scientists discovered a herd of unicorns
         living in a remote valley."

greedy: " The unicorns were so rare that they were not even recognized by
         the local people. The unicorns were so rare that they were not
         even recognized by the local people. The unicorns were so rare
         that they were not even recognized by the local people. ..."

         repeated 4-grams: 87.6 %

Eight sentences, one sentence. Nearly nine out of ten four-token windows had already appeared earlier in the same output. This is neural text degeneration, named and explained by Holtzman et al. in the paper that introduced top-p.³ The model is not broken; maximising sequence probability is simply the wrong objective for open-ended text. Human writing is not the most likely sequence of words — it carries surprise, its per-token probability wandering, dipping and recovering — while the maximum-probability path is a fixed point that, once entered, has no reason to leave.

That is why sampling exists at all. It is also, and this is the part that gets left out, not a universal law. Chapter 12 measured 24 out of 24 correct on two-step word problems with plain greedy decoding, and sampling at temperature 0.8 dropped that to 81 %; self-consistency then spent six times the tokens climbing back to where greedy already was. Both facts are true at once:

Most production prompts are the second kind and get configured like the first, because the temperature was left at whatever the example code used.

Sampling from the full distribution is not what anyone actually does, because the tail is enormous and full of nonsense. Something has to be cut. There are two classical answers and they differ in one respect that decides everything.

Top-k keeps a fixed number of candidates. Sort by probability, keep the first $k$, discard the rest, renormalise.⁴ Top-p, also called nucleus sampling, keeps a fixed amount of mass: take tokens in descending order until their cumulative probability reaches $p$, and stop.³ Formally, the nucleus is the smallest set $V_p$ with

The difference sounds cosmetic and is not, because the two prompts you send in the same minute have completely different distribution shapes. Both of these are the same model at temperature 1:

One fixed $k$, two failures in opposite directions. On the factual prompt, $k = 40$ admits 39 tokens that between them are worth 3.6 % — it is letting rubbish through, including a candidate at nine thousandths of a percent, because the rule counts slots and not evidence. On the story prompt, the same $k = 40$ throws away 21 % of the mass the model genuinely assigned, because the real nucleus there is 467 tokens wide.

Top-p makes exactly one number do both jobs. Set $p = 0.9$ and it keeps 1 token on the first prompt and 467 on the second, because it is asking a question about the distribution rather than imposing a count on it. Watch that adaptation directly — same cut, four temperatures:

That widget also settles a misconception worth naming, because it costs people real money. On a confident distribution, top_p = 0.9 is not "a bit of variety". It is greedy. At temperature 1 the leading token here holds 96.90 %, which is already over 0.9, so the nucleus is one token wide and nothing else can ever be drawn. Teams set top_p to 0.9 believing they have loosened something and then wonder why every response is identical.

Set top-k instead and the opposite failure is just as visible:

Three different mechanisms travel under similar names, they do different things, and the difference is measurable. Let $c_i$ be the number of times token $i$ has already appeared.

The same degenerate continuation from earlier, with each one applied. "Steps altered" counts how many of the 120 generation steps picked a different token than the unpenalised model would have. The run is 120 steps here against 140 in the block above, which is why the unpenalised baseline reads 85.5 % rather than 87.6 %:

Three things fall out. Presence at 0.5 changed three decisions out of 120 and cut repetition by a quarter — the loop was held together by a handful of tokens. Frequency at 0.5 changed four times as many decisions for a far larger effect, because the count multiplier keeps growing while the presence constant does not. And the CTRL penalty at the widely-copied value of 1.2 rewrote 35 of 120 decisions, which is not a nudge; it is a different model.

That last number is the setup for the failure nobody warns about.

Code repeats. Tables repeat. Lists repeat. Structured output repeats by definition — that is what structure is. A penalty cannot tell the difference between a model stuck in a loop and a model correctly emitting the fourth row of a table, because both look like a token appearing again.

The same three tasks, generated three ways:

The frequency penalty at 0.5 turned out to be harmless on all three, which is a useful and slightly surprising result, and it says something precise: since no decision changed, the structural tokens must have been winning their positions by more than the penalty subtracted, even after appearing five and six times. The CTRL penalty, which divides instead, does dislodge them, and here is what it produced:

repetition 1.2, markdown table:
  | n | 2^n |
  | --- | --- |
  | 0 | 1      |
  | 1 | 2       |
  | 2 | 4       |

The alignment falls apart: the amount of padding inside each cell changes from row to row, because the run of spaces before the closing pipe is exactly the kind of repetition the penalty exists to break. Cosmetic, and it cost six extra tokens. The Python case is not cosmetic:

nothing / frequency 0.5:
      total = 0
      for i in range(1, n + 1):
          total += i ** 2
      return total

repetition 1.2:
      # Initialize total_sum with 0
      total_sum = 0
      # Loop through numbers from 1 to n, incrementing by 2 each time
      for i in range(1, n + 1,

The penalty pushed the model off total — already used in the docstring — onto total_sum, padded the output with invented comments to spend its budget on unused tokens, and then walked into a three-argument range with a stride. The comment says incrementing by 2 each time, which is wrong for a sum of squares from 1 to $n$. A repetition penalty produced incorrect code from a prompt that was answered correctly without it.

The rule that follows is short: penalties are for open-ended prose, and they should be off for code, structured output, tabular data and anything with a schema. Chapter 18 is about exactly that second category.

Every real implementation applies these in one specific sequence:

penalties → temperature → top-k → top-p → sample

This is not arbitrary bookkeeping, and swapping two stages produces genuinely different distributions. Two measurements, both on the factual prompt.

Cutting before or after the temperature. The nucleus is computed on whatever distribution it is handed, and temperature changes that distribution radically:

At $T = 2$ the same nominal setting yields a candidate set of 32,966 or of 1, depending purely on which stage runs first. If you have ever wondered why raising the temperature "does nothing" on one provider and destroys the output on another with the same two numbers, this table is a plausible answer.

Penalising before or after the temperature. Subtracting a penalty $\alpha$ and then dividing by $T$ gives an effective penalty of $\alpha/T$; dividing first and then subtracting gives $\alpha$. With a presence penalty of 1.0 applied to the leading token:

Identical at $T = 1$, as they must be. A factor of 1.58 apart at $T = 2$. "Presence penalty 1.0" is not a well-defined amount of penalty unless you also know where the temperature is applied, and no API documents this.

def sample(logits, counts, presence=0.0, frequency=0.0,
           temperature=1.0, top_k=0, top_p=1.0, generator=None):
    z = logits.clone()

    idx = torch.tensor(list(counts))                       # 1. penalties
    if len(idx):
        z[idx] -= presence
        z[idx] -= frequency * torch.tensor([float(c) for c in counts.values()])

    if temperature <= 0:                                   # 2. temperature
        return int(z.argmax())                             #    T=0 is argmax
    p = torch.softmax(z / temperature, -1)

    p, order = p.sort(descending=True)
    if top_k:                                              # 3. top-k
        p[top_k:] = 0
    p = p * ((p.cumsum(0) - p) < top_p)                     # 4. top-p

    p = p / p.sum()                                        # 5. renormalise
    return int(order[torch.multinomial(p, 1, generator=generator)])

Every provider takes a different subset of these controls, with different ranges, and ignores the rest in silence. This is not an abstract complaint. Any application that offers a choice of model has to write the differences down somewhere, and the file where it does is a map of the incompatibility. Here is what one such catalogue declares for a single parameter across the nine text sources it supports:

The word is the same; the scale is not. A "temperature of 1" is the unmodified distribution on one and the maximum permitted heat on another, and half the catalogue cannot express the value the other half treats as neutral-plus-a-bit. The rest of the knobs are just as uneven: the OpenAI, DeepSeek and xAI entries take presence and frequency penalties and no topK; the Google, Meta, Cerebras and PaLM entries take topK and no penalties; Anthropic takes topK, topP and stop sequences and no penalties; and exactly one of the nine — Mistral — takes a seed. Sending a parameter a provider does not implement generally produces no error at all: the request succeeds, the knob does nothing, and you conclude the setting has no effect.

And notice what such a file is: a claim about somebody else's API, written on one particular day, that nothing verifies afterwards. A catalogue that says 0 to 1 for a provider that now accepts 0 to 2 will silently cap every request.

Two more controls belong to the same family. logprobs, where offered, returns the log-probabilities of the chosen token and often the top few alternatives — the only window you get onto the distribution this chapter is about, and the basis of every confidence heuristic built on a closed model. And maximum tokens plus stop sequences end generation without reference to probability at all: a hard cap and a string match. Both surface as the finish_reason from Chapter 14, where length means your answer was cut mid-sentence by a budget, not finished by the model.

Set a seed and sampling becomes reproducible. That part is real, and it is easy to verify:

seed = 1234  " Paris\nWhat is a good geographical qualifier for describing
               Paris concerning its location?\nA: Near the Mediterranean Sea"
seed = 1234  " Paris\nWhat is a good geographical qualifier for describing
               Paris concerning its location?\nA: Near the Mediterranean Sea"
seed = 7     " Paris is the capital of France. The appellation of Paris is
               \"Île de Paris\"."
seed = 7     " Paris is the capital of France. The appellation of Paris is
               \"Île de Paris\"."

Byte-identical within a seed, different across seeds, exactly as advertised. So what the seed fixes is the random draw in the last line of that sample function — which token gets picked given a distribution.

What it does not fix is the distribution. And that is where the trouble is, because the vector of logits your model produces is not a mathematical object; it is the output of billions of floating-point additions, and those have an order.

Chapter 2 left this experiment ready. The same million float32 numbers, summed in different groupings:

sequential          998.564270020    error vs float64: 6.393e-03
pairwise (numpy)    998.570556641    error vs float64: 1.061e-04
in 4 chunks         998.570495605    error vs float64: 1.672e-04
in 8 chunks         998.570556641    error vs float64: 1.061e-04
in 16 chunks        998.570678711    error vs float64: 1.594e-05

sequential == pairwise?  False
4 chunks == 8 chunks?    False

Look at the last line. The number of chunks changes the answer. That is not a curiosity about numpy; it is the mechanism, because when an inference server splits a reduction across more or fewer parallel units, it is doing precisely this. And a server splits according to how many requests it is serving.

Here is that effect on the model itself. The same prompt, the same forward pass, the only difference being how many other requests happened to be in the batch:

20 identical forward passes, batch of 1:  20 / 20 bit-for-bit identical

the same prompt inside a batch of  2:  147,321 of 151,936 logits differ
the same prompt inside a batch of  4:  146,515 of 151,936 logits differ
the same prompt inside a batch of  8:  146,515 of 151,936 logits differ
the same prompt inside a batch of 16:  147,321 of 151,936 logits differ

largest change to any logit: 2.5e-05

Run alone, the model is perfectly deterministic — twenty passes, identical to the bit. Put the identical prompt in a batch with unrelated requests and 97 % of its logits change. Nothing about your request changed. Somebody else's request arrived.

Now the honest part, because this is usually told as though it were the end of the story. A change of $2.5 \times 10^{-5}$ only alters the output if two candidate tokens were within that of each other. Over 717 generation steps across twelve prompts, the smallest gap between the top two logits was $2.5 \times 10^{-3}$ — a hundred times larger than the perturbation — and no step was close enough to flip. So on this model, in float32, on a laptop, batching moved every logit and changed no token.

That is a description of favourable conditions, not a reassurance, and one change to those conditions is enough:

same weights, same prompts, greedy decoding, no seed involved
float32 vs bfloat16:   6 of 8 answers diverge
                       first divergence at step 23, on average

  float32: "...it is scattered and dispersed into different colors,
            including blue. The blue light is scattered more than other
            colors, so it appears to come from the sky."

  bfloat16: "...it is scattered and scattered, causing the colors of the
             sun to be scattered and scattered, creating the appearance
             of a blue color."

Six of eight answers diverge, and one of them degrades badly. Chapter 2's table says why: bfloat16 keeps 7 mantissa bits, so near a logit magnitude of 16 the representable values are 0.125 apart — 16.0, then 16.125, then 16.25 — and rounding can move a logit by up to 0.0625. Meanwhile 4.7 % of the generation steps measured above had a top-two gap below 0.1. That is the whole difference between the two experiments: in float32 the perturbation was a hundred times smaller than the closest decision, and in bfloat16 it is the same size. Production inference runs in 16-bit, on hardware with fused kernels and reduction orders nobody promises to keep. Whether "the numerical noise is negligible" is a question about precision and hardware, not about the model.

So, the four causes, catalogued as Chapter 9 promised:

OpenAI's seed parameter is honest about this in the only way it can be: it ships alongside a system_fingerprint field identifying the backend configuration, and the documentation states that determinism is best-effort and that a changed fingerprint means results may differ. Read that as what it is — a provider telling you that it controls all four causes above, that you control none of them, and that the one thing it can offer is to tell you after the fact that something moved.

Everything here has been about a knob and its consequences. Step back one level and the harder problem appears: the object we have been tuning is a probability distribution, and probability distributions do not have an interface.

A function call has one. A database row has one. A POST handler expecting a JSON body with three required fields has one, and it will reject anything else. Between the model and every other component in your system sits a contract that one side cannot make promises about: the model will produce something, drawn from a distribution you have shaped but not fixed, and the code on the other side needs a value of a known type or it throws.

The bridge between those two worlds is built from this chapter's material rather than from parsing and retries. If a token would break the required structure, you do not sample it and hope — you set its logit to $-\infty$ before the softmax ever sees it. Constrained decoding is a mask over the same vector we have spent this chapter reshaping, and it turns "please reply in JSON" from a request into a guarantee.

Chapter 18 is that contract: tool calling, JSON Schema, structured outputs, and what it takes to make a deterministic system safe to build on top of a probabilistic one.

All measurements in this chapter come from Qwen/Qwen2.5-0.5B-Instruct on CPU, float32 unless stated, with the sampling implemented as written in the optional section rather than delegated to a library. They are a small model, and the specific values are its; the mechanisms are not. Von Platen's How to generate text with different decoding methods (Hugging Face, 2020) is the article this one is measured against and is still the best short introduction to the same material. For the determinism section: PyTorch's reproducibility notes describe what a seed does and does not fix on a single machine, OpenAI's documentation of seed and system_fingerprint describes what a provider can and cannot promise, and Thinking Machines' 2025 discussion of batch-invariant kernels is the clearest public account of why fixing this at the inference-server level is possible but not free.