Byte-level tokens: how AI can read any language, symbol or emoji

What byte-level tokens are

Computers do not store letters. They store numbers. Every character you type, from "A" to "é" to a Chinese character to an emoji, is saved as a short run of bytes: small numbers that can only take 256 different values. Byte-level tokens mean that an AI's text chopper, its tokenizer, starts from exactly those 256 values. Everything else it learns is built on top of them.

What the video shows

The video explains that every character is stored as one or a few bytes, and that a byte can take only 256 possible values. AI tokenizers start from those 256. The result: no text, language or emoji is ever impossible for the model to read.

How characters become bytes

Each character has an official number in a worldwide standard called Unicode, which covers about 149,000 characters. "A" is number 65 and "é" is number 233. A standard way of saving those numbers, called UTF-8, spends a different number of bytes depending on the character:

| Character | Bytes in UTF-8 | |---|---| | An English letter, like "A" | 1 | | "é", or a Russian (Cyrillic) letter | 2 | | Most Chinese and Japanese characters | 3 | | An emoji | 4, often more |

Many emoji take eight bytes or more, because family emoji and skin tones are built by combining several pieces with special joining characters.

Why start from bytes?

Why not simply give every Unicode character its own entry in the tokenizer's list instead? Because that list would need around 149,000 entries, and most of them would never appear in the model's training text. Bytes are the opposite: a starting list of exactly 256, and every possible text can be written with them. There is never an "unknown" character, because there is nothing that cannot be spelled out byte by byte.

An everyday example

Think of Morse code. It has only two signals, dots and dashes. Each letter is a short run of them, and with those runs you can spell out any word, including names nobody had heard of when the code was invented. Bytes play the same role for a tokenizer: every character is a short run of them, so a tiny set of building blocks can spell any text. On top of them, the tokenizer learns shortcuts for byte combinations that show up often, so common words do not have to be spelled out byte by byte every time.

Two consequences worth knowing

• The model reads bytes, not text. What it really processes are byte sequences that happen to turn into text on your screen. That is why the word "café" can be saved in two different ways that look identical to you, yet are different bytes to the model and get split into different pieces.
• Not every language gets the same deal. If the tokenizer learned its shortcuts mostly from English text, other writing systems get fewer shortcuts and need more pieces. An English letter is one byte; a typical Japanese character is three. That uneven cost gets its own video in this series: the token tax.

Why it matters

Starting from bytes is the reason a modern chatbot can accept any language, any symbol and any emoji without breaking. It is also the root of a few quirks you may run into: text that looks the same to you may not be the same to the model, and some languages simply cost more pieces than others. Knowing that the model works on bytes underneath makes both of those less mysterious.

Learn it step by step in Chapter 7 of our free course AI From Scratch: Build a BPE Tokenizer: Why Your Model Can't Count the R's.