Self-supervised learning: how text becomes its own answer key

What self-supervised learning is

A lot of classic AI training needs a person to supply the right answers. Someone tags thousands of photos as "cat" or "dog", or marks emails as spam, and the model learns from those labels. Self-supervised learning skips that step. The answers are already inside the data, so the data does the teaching.

Language models, the kind of AI behind chatbots, are the clearest case. In their main training phase they do one job: guess the next token (a word or a piece of a word) given everything that came before it. And the correct answer for that guess is sitting right there in the text. It is simply the token that comes next.

What the video shows

The video starts from a puzzle: nobody ever labeled the internet, so how did chatbots learn from it? The answer is that they never needed labels. In any piece of text, the right answer for each spot is just the next word. That turns every sentence into a set of free practice questions, and the whole internet into one giant workbook.

An everyday example

Imagine learning a language by reading a novel with a card covering the page. You read up to the card, guess the next word, slide the card along and check. Nobody has to sit beside you with an answer sheet. The book itself tells you whether you were right.

One short sentence already holds several lessons. Take "the cat sat on the mat":

• after "the", guess "cat";
• after "the cat", guess "sat";
• after "the cat sat", guess "on";

and so on to the end. The answers are the same sentence, moved along by one position.

Why the answers come for free

This is the part that changed the economics of AI. Labeling by hand is slow and expensive, so any dataset built by people stays fairly small. Text needs no labeling at all: a trillion tokens of text are a trillion ready-made training examples. That is why a modern model learns from something close to "the internet" instead of a collection that somebody put together by hand. The course calls this the economic fact underneath the whole field.

The scoring is just as simple. At every position, the model is judged on how much probability it gave to the token that really came next. More probability on the true next token means a better score.

A common misconception

People often dismiss chatbots with "it only predicts the next word". The course argues that the logic runs the other way. If a model gives odds for each next token, multiplying those odds along the text gives the probability of the entire document, exactly. Predicting the next token is not a shortcut around modelling text. It is the full task, with nothing left over.

Why it matters

Self-supervised learning explains why language models could be trained on such enormous amounts of text. The supply of lessons was limited by how much text existed, not by how many people were available to label it. Once you see that the labels are just the data shifted by one step, the rest of how these models learn starts to make sense.

Learn it step by step in Chapter 8 of our free course AI From Scratch: Next-Token Prediction: Embeddings, and What Perplexity Means.