Tokenization

Tokenization is the first step every large language model takes with your text. Before a model can predict a next word, it must break your input into pieces it can look up in its vocabulary. This step silently decides three things that matter to you as a builder: the cost of every API call, how well the model handles languages other than English, and even whether it can do arithmetic correctly. Understanding tokenization helps you predict costs, debug unexpected model behavior, and make informed choices about prompts and model selection. It unlocks the study of Byte-Pair Encoding, Context Window, and the inner workings of Large Language Models.

Imagine you are writing a message to a friend, but you are only allowed to send it in pieces, and each piece has a fixed label. The model works the same way: it does not read your text as characters or whole words. Instead, it has a fixed list of allowed pieces, and it must find a way to break your text into those pieces. This process is called tokenization. A token can be a whole word ('apple'), a part of a word ('app' and 'le'), or even a single character if nothing else fits. The model's vocabulary is the list of all allowed tokens, and each token has an ID number. When you send text to a model, it first converts it into a sequence of these IDs, and that sequence is what the model actually processes. This is not a detail you can ignore: the same text can be tokenized in many ways, and the choice affects how much the model 'sees' and how much it costs. For example, the sentence 'I love biryani' might become 5 tokens: 'I', ' love', ' bir', 'yani'. The model does not see spaces or punctuation; it sees these tokens in order. This is the first real step in how a model understands language.

Think of tokenization like packing a suitcase for a flight. You have a fixed set of luggage sizes the airline allows (these are your tokens). Some items fit in a single bag (common words like 'the'), but a bulky item (an uncommon word like 'supercalifragilistic') must be broken into smaller pieces that each fit. The airline's list of allowed bag sizes is the vocabulary, and you must use only those sizes. You cannot bring a bag that is not on the list. If a word is so rare that no piece of it fits, you might break it down to single letters, because those are always allowed (like a 'fragile' sticker) – this is the fallback to bytes. The packing process is the tokenizer. A good tokenizer is like an efficient packer: it uses the fewest bags possible for common words, but it never fails to pack anything, because single characters or bytes always fit. Where the analogy stops working: in a suitcase, the order of bags doesn't matter to the airline, but the model cares about the order of tokens because that order carries the meaning of the sentence. Also, a suitcase has a physical limit for the total weight, but a model's context window limits the number of tokens, not their total 'weight' – and that limit is where cost and memory come from.

Tokenization is the process of breaking text into a sequence of discrete units (tokens) from a fixed vocabulary, which the model then maps to integer IDs for processing.

This is the entry point to understanding how large language models work. It connects directly to the Context Window (the maximum number of tokens a model can attend to), Byte-Pair Encoding (the algorithm that chooses the vocabulary), and Token IDs and Special Tokens (how tokens map to numbers and control markers). It also connects to the broader concept of Foundation Models, because the tokenizer is baked into a pretrained model and cannot be changed later. Understanding tokenization is essential for the Quality-Cost-Latency Tradeoff: the number of tokens directly determines cost and latency. Your library notes emphasise that tokenisation 'silently determines cost, multilingual quality and arithmetic ability' – a key insight you should hold onto.