Encoder vs decoder: why chatbots kept only half of the transformer

What an encoder and a decoder are

The transformer is the design behind today's language models. The original one, from 2017, was built for translation, and it came in two halves:

• The encoder is the reader. It takes in the whole source sentence at once, and every word can look at every other word, before and after it.
• The decoder is the writer. It produces the translation one word at a time, and each new word can only look at the words already written.
• A connecting layer, called cross-attention, lets the writer look back at what the reader understood while it writes.

That split makes sense for translation, where the input and the output are two separate texts.

What the video shows

The short sums up the story simply. The first transformer had a reader that sees the whole text and a writer that goes word by word. The chatbots we use today kept only the writer, because a writer on its own can learn from any text at all.

An everyday example

Imagine an interpreter who waits for the speaker to finish each sentence. First they take in the whole sentence. Then they say it in another language, word by word, checking back against what they heard. That is the encoder-decoder shape: one part that listens to everything, one part that speaks in order, and a link between them.

Now imagine a writer who can only ever continue the page in front of them. Hand them a page that reads "English: Good morning. French:" and the natural next words are the translation. Hand them a question, and the natural next words are its answer. That is the decoder-only shape.

Why the writer won

The course is clear that it was not about elegance. It was about data. A decoder-only model learns one simple task: predict the next token, a small piece of text that is often a word. That task works on any text whatsoever, so the model can learn from the whole internet instead of needing matched pairs of the same text in two languages. And almost every job can be written as a document to continue:

• a translation is the source text followed by its translation;
• a question followed by its answer is a short document;
• even a conversation where the AI calls a tool halfway through is, in the end, one document.

One stack of layers, reading in one direction, doing one task, with far more material to learn from.

Where encoders still win

Encoders did not disappear. When the job is to describe a whole text rather than to continue it, seeing everything at once is exactly what you want. That is why, later in the course, the embeddings used for search (a list of numbers that captures what a text is about, so similar texts can be found) come from an encoder, and not from the model doing the chatting.

Why it matters

Knowing which half you are dealing with tells you what a model is built for. A decoder is built to continue a text, which is exactly what a chatbot does when it replies. An encoder is built to take in a whole text and sum it up, which makes it the natural choice for finding the right documents. A system that first looks up documents and then answers can use one of each: an encoder to find, a decoder to write.

Learn it step by step in Chapter 9 of our free course AI From Scratch: Attention and the Transformer Block, Derived From an Average.