Byte-Pair Encoding

Byte-Pair Encoding is the first piece of the tokenisation pipeline that every large language model — including ChatGPT-style models and small on-device ones — uses to turn text into numbers the model can read. Without it, you cannot understand why your prompt costs a certain number of tokens, why some languages are cheaper than others, or why the model struggles with arithmetic. Master this, and you unlock your library's notes on Tokenization, Token IDs and Special Tokens, the Context Window, and ultimately the entire Large Language Models topic. It is the quiet foundation under everything else.

Think of the English language as a set of building blocks. You could use single letters — 'c', 'a', 't' — and spell every word as a sequence of letters. That works, but it is wasteful: the word 'the' would need three blocks every single time you write it. At the other extreme, you could use whole words as blocks — 'the', 'cat', 'sitting' — but then any word you have never seen before, like a new name or a typo, has no block and you are stuck. Byte-Pair Encoding finds a middle ground. It starts with single letters (technically bytes, but think letters for now) and then looks at your entire training corpus — all the text the model will learn from — and finds which two blocks appear together most often. It merges those two into one new block. Then it repeats: find the most frequent adjacent pair among the current set of blocks, merge them, repeat. After many rounds, you have a vocabulary of blocks: some are whole frequent words like 'the', some are common parts like 'ing', and some are single letters. When you later see a new word like 'unalone', you can break it into pieces you already have: 'un', 'al', 'one' — or if nothing matches, down to single letters. So no word is ever unknown: the worst case is you spell it letter by letter. This is exactly the intuition from your library: 'Frequent words become single tokens; rare ones decompose into pieces.' The magic is that the merges are learned from data, not chosen by a linguist, and after training they are frozen — the model never changes its vocabulary again.

Imagine you are a librarian who has to file every sentence as a sequence of cards. Initially, each card holds one letter. You notice the cards 't' and 'h' appear together very often, so you create a new card 'th' and replace every adjacent 't','h' pair with this single card. That saves space and speeds up your filing. Then you notice 'th' and 'e' together are common, so you make a 'the' card. Over time, you build a drawer of cards: some are whole words like 'the', some are parts like 'ing', and some are single letters. When a new document arrives with a word you've never filed before, like 'xylophone', you can always spell it with single-letter cards — 'x','y','l','o','p','h','o','n','e' — so you are never stuck. The analogy stops working at one point: in reality, the merges are decided purely by frequency in a large corpus, not by any sense of meaning. The librarian might merge 'q' and 'u' because they are frequent, but she might also merge 'e' and 'r' if they happen to appear together often, even though 'er' is a meaningful suffix. There is no grammar in the process — just statistics. Also, the librarian would never merge letters across word boundaries (like 'e' and space) because spaces are handled separately in modern tokenisers, but the principle is the same.

Byte-Pair Encoding (BPE) is an algorithm that builds a token vocabulary by starting with individual bytes and repeatedly merging the most frequently occurring adjacent pair in a training corpus, until the vocabulary reaches a target size.

This concept connects directly to your library's notes on Tokenization (the process of splitting text into units), Token IDs and Special Tokens (each BPE token gets an integer ID), and the Context Window (the number of tokens limits the window size). It is the first step in the Large Language Models pipeline, before embeddings and attention. You have just learned this from scratch, so this fills the gap between raw text and the model's input.