AI glossary
Tokenization
Tokenization is the process of splitting text into smaller units, called tokens, that a language model processes as its basic units of input and output, since the model works with numbers rather than raw text directly.
Before a large language model can understand or generate text, that text must be converted into a format the neural network can compute. A tokenizer handles this conversion by breaking input text into tokens and assigning each unique token a specific numeric ID. The model never sees words or characters; it only sees these IDs. This mapping between text and numbers is the foundation of how natural language processing systems interpret human language.
Why models need tokenization
A language model cannot process raw characters or words directly as text; every piece of input and output must first be converted into a sequence of numeric IDs the model can compute with. These IDs act as indices into a vocabulary of known tokens. When you send a prompt to an AI, the tokenizer splits your text into these tokens, and the model processes the resulting sequence of numbers through its layers to produce a prediction.
The output is also generated token by token. The model predicts the next token ID based on the previous ones, and the tokenizer converts that ID back into human-readable text. This process underpins the transformer architecture, which relies on attention mechanisms to weigh the importance of each token relative to others in the sequence. Without tokenization, the model would have no standardized way to represent the infinite variety of human language in a fixed numerical space.
Subword tokenization
Tokens can be whole words, parts of words (subwords), or individual characters, depending on the scheme; most modern large language models use subword tokenization. This approach strikes a balance between efficiency and coverage. Instead of requiring a unique token for every possible word, subword tokenization builds a vocabulary of common word pieces from a large body of training text.
This method solves the “out of vocabulary” problem faced by older whole-word tokenizers. If a whole-word tokenizer encounters a word it hasn’t seen before, it often marks it as unknown or misses it entirely. A subword tokenizer can always represent any word by breaking it into smaller, more common pieces already in the vocabulary. For example, the word “unhappiness” might be split into “un”, “happi”, and “ness”. This allows the model to handle rare or complex words effectively, as it can reconstruct them from known components. The result is a more flexible system that generalizes better across diverse texts.
Common tokenization algorithms
Several algorithms drive modern tokenization, with Byte Pair Encoding (BPE) being one of the most widely used. BPE builds its vocabulary by starting with individual characters and repeatedly merging the most frequently occurring adjacent pairs into new subword units. It continues this merging process until a target vocabulary size is reached. This greedy approach ensures that the most common sequences are represented efficiently, reducing the total number of tokens needed for common text.
Another notable toolkit is SentencePiece, which can implement BPE or a related algorithm. Unlike traditional tokenizers that assume text is already split into words, SentencePiece treats the input text as a raw stream of characters, including spaces. This makes it particularly well-suited for languages that do not use spaces between words, such as Chinese, Japanese, or Thai. By learning segmentation directly from raw text, these algorithms work across multiple languages without requiring language-specific preprocessing rules.
Why tokenization affects cost
Most commercial LLM APIs price usage per token rather than per word or character, so the way an input or output is tokenized directly determines the cost of a request. A single word can become multiple tokens if it contains complex morphology or rare characters. For instance, a long compound word might be split into several subword units, each incurring a small fee. Understanding how your text is tokenized helps you estimate costs more accurately.
Context window limits, the maximum amount of text a model can process at once, are also measured in tokens rather than words or characters. How efficiently a tokenizer represents a piece of text affects how much content actually fits. If a tokenizer splits simple text into many small tokens, you might hit the context limit faster than expected, cutting off important information. Efficient tokenization maximizes the usable context window, allowing you to pass more data to the model within the same budget. This is why developers often monitor tokens usage closely during development and production.
Limitations
Tokenization is not perfect, and it can behave unevenly across languages. Languages with vocabularies and structures well represented in a tokenizer’s training data are often tokenized more efficiently (fewer tokens per word) than languages that are less represented. This disparity can affect both cost and the effective context length available for non-English texts. If a language is underrepresented, its words may be split into many small pieces, inflating token counts and costs.
Numbers, code, and unusual formatting can sometimes be split into tokens in ways that are not intuitive to a human reader. This occasional fragmentation can affect a model’s ability to handle precise numeric or structured tasks. For example, a date or a specific code identifier might be broken in a way that obscures its meaning to the model. Being aware of these limitations helps you design better prompts and preprocess data appropriately, ensuring the embedding of semantic meaning remains intact despite the mechanical splitting of text.
FAQ
What is tokenization in AI?
Tokenization is the process of splitting text into smaller units called tokens, which are converted into numeric IDs for a language model to process. It is the first step in converting human language into a format that neural networks can compute.
How does subword tokenization work?
Subword tokenization breaks text into parts of words rather than whole words. It uses common word pieces to represent both frequent and rare words, solving the “out of vocabulary” problem by allowing any word to be constructed from known subword units.
Why does tokenization matter for cost?
LLM APIs charge per token, not per word. Since one word can become multiple tokens depending on the tokenizer, efficient tokenization directly impacts your bill and how much text fits into the model’s context window.
What is Byte Pair Encoding (BPE)?
BPE is an algorithm that builds a vocabulary by starting with characters and merging the most frequent adjacent pairs into new units. It continues until a target vocabulary size is reached, optimizing for efficiency in representing common text sequences.