Chat templates: the hidden markers that tell an AI whose turn it is

What a chat template is

When you chat with an AI, you see a tidy conversation. The model sees something else: one long stream of text, with special markers wrapped around every message. A chat template is the fixed pattern of those markers. They say who is speaking (the system instructions, the user or the assistant) and where each turn begins and ends.

In one common format, a user's message is wrapped like this: <|im_start|>user … <|im_end|>. Those markers are not ordinary characters that happen to look odd. Each one is a real entry in the model's vocabulary, the fixed list of text pieces it can read and write. They were added before the model was fine-tuned into an assistant, and during that training it met them millions of times, always in the same spots.

What the video shows

The video is about the hidden markers that sit around each message in a chat, marking who is talking and where a turn ends. Its point is simple: leave them out and the very same AI quietly gets worse.

An everyday example

Think of two people on walkie-talkies. Each one ends with "over" so the other knows it is their turn to speak. Drop the "over" and they talk across each other, or both wait in silence. The words are the same, but the conversation stops working. The chat template is the model's "over", plus a name tag on every line.

What happens without it, measured

The course runs a simple test on a small model, Qwen2.5-0.5B, which has half a billion parameters (the numbers it learned during training). Same question, "What is the capital of France?", and greedy decoding, which means the model always picks its single most likely next piece of text, so luck plays no part.

• With the template, it replies "The capital of France is Paris." and stops.
• Without it, sent as a bare question, it gives the same correct answer, then announces that it will double-check with a short Python program, and starts writing the code.

The answer is right both times. The difference is behaviour. A bare question looks like nothing the model practised on while it was being fine-tuned, so it drifts into doing something nobody asked for.

How to spot it

This mostly affects people who send text straight to a model, for example when running an open model themselves, rather than using a finished chat app that handles the template for them. Warning signs:

• The model answers, then keeps going with something unrelated.
• It seems noticeably worse than the same model elsewhere, such as in its maker's own chat app.
• Nothing crashes and no error message appears.

That last point is what makes it dangerous. A missing or wrong template never announces itself; it just lowers the quality of every answer. According to the course, it is the most common reason people complain that a model "got dumber" once they called it directly. The fix is to use the exact template that particular model was trained with, because formats differ between models.

Why it matters

Treat the template as part of the model itself, not as wrapping paper around it. The model learned to behave inside those markers, and outside them it is working on unfamiliar ground. A silent loss of quality is harder to catch than an error, which is why it is worth checking first whenever a model disappoints.

Learn it step by step, with the full with-and-without test, in Chapter 11 of our free course AI From Scratch: From Base Model to Assistant: SFT, RLHF, DPO and GRPO.