SentencePiece: how AI splits text in languages without spaces

What SentencePiece is

SentencePiece is a tool for building and running a tokenizer, the program that cuts text into chunks (tokens) before a language model reads it. According to the course, it is the tokenizer most non-English language models are built with.

Its big idea is simple: spaces get no special treatment. SentencePiece reads the text as one raw stream of characters, and it turns the space into an ordinary visible symbol, just like a letter.

What the video shows

The video contrasts two approaches. Many tokenizers split text on spaces first, and that does not work for languages like Japanese, where words are not separated by spaces. SentencePiece treats the space as just another character, so every language is handled the same way.

An everyday example

Try reading this: "nowisthetimetolearn". You can untangle it because you know the words. But a rule that says "find the spaces, then cut there" has nothing to work with.

That is the normal situation in Japanese and Chinese, which are usually written without spaces between words. A tokenizer that counts on spaces to show where one word ends and the next begins is relying on something those languages simply do not provide.

Now imagine writing English with each space swapped for a visible mark, say an underscore: "_good_morning". The space is no longer a gap to split on. It is one more character that can end up inside a piece, like any letter. That, roughly, is what SentencePiece does.

How it works

• No splitting step first. Some tokenizers, like GPT-2's, first chop the text into word-sized chunks with a fixed pattern and never build a piece across those cuts. SentencePiece skips that step entirely and works on the whole stream.
• The space becomes a symbol. Because the space is stored as a visible character, it can sit inside a token. When the pieces are glued back together, the spaces come back exactly where they were.
• Two engines underneath. SentencePiece can run BPE, which builds a vocabulary by repeatedly gluing together frequent pairs, or Unigram, which works the other way round: it starts from a big list of candidate pieces and keeps removing the ones the text would miss least.

Unigram has one extra trick. It gives a probability to each possible way of cutting up a string, so during training a model can be shown several different cuttings of the same text. That acts as a training aid, and stops the model from leaning too hard on one exact split.

Do we even need words?

It is easy to assume the word is the natural unit of text. Yet in some writing systems, where one word stops and the next begins is not marked at all. SentencePiece sidesteps the question: it never needs to know where the words are.

Why it matters

Not every writing system puts spaces between words. A tokenizer that assumes spaces fits some languages neatly and others awkwardly. By making the space just another character, SentencePiece gives every language the same starting rules. That helps explain why, according to the course, most models built for languages other than English use it.

Learn it step by step in Chapter 7 of our free course AI From Scratch: Build a BPE Tokenizer: Why Your Model Can't Count the R's.