AI glossary
Tokens
A token is the basic unit of text that a language model actually processes. It is a chunk of text produced by a tokenizer that may represent a whole word, a part of a word (subword), a single character, or even punctuation, depending on the specific rules of the underlying tokenization system.
Tokens vs words and characters
Tokens are not the same as words. While a simple word count might suggest a certain length, the number of tokens can differ significantly. Common, short words are often represented as a single token. In contrast, longer or less common words are frequently split into multiple subword tokens.
Some tokens may only represent a fragment of a word, while others may include a leading space as part of the token itself. This distinction is critical because it means that text length in characters or words is an imperfect proxy for how much data the model actually consumes. Understanding this difference helps you estimate processing needs more accurately than relying on word counts alone.
How tokenizers split text (subword tokenization)
Most modern large language model architectures rely on a subword tokenization scheme, such as byte pair encoding (BPE) or similar algorithms. These systems build a fixed vocabulary of common word pieces. They start with individual characters and iteratively merge the most frequently occurring adjacent pairs into new vocabulary entries.
This subword approach allows a model to represent virtually any text, including words it never encountered during training. It does this by falling back to smaller, familiar pieces or, in the worst-case scenario, individual characters. This prevents the model from failing on unknown words, which would happen with a strict whole-word vocabulary. The transformer architecture that powers these models depends heavily on this efficient representation to handle diverse linguistic inputs.
Roughly how many tokens per word
When estimating costs or context limits, it helps to have a general sense of how text converts to tokens. OpenAI’s public documentation on tokenization provides a widely cited rule of thumb for English text. It suggests roughly 4 characters per token, or approximately 3 tokens for every 4 words.
However, this is just an average. The exact count for any specific piece of text depends on the tokenizer being used. Variations arise from punctuation, capitalization, and how common each word is in the model’s training data. Rare words or complex technical jargon may split into more tokens than common language, affecting both processing time and cost.
Context windows
A model’s context window defines the maximum number of tokens it can take into account at one time. This limit covers both the input prompt you send and the model’s own generated output combined. If your total token count exceeds this window, the model simply cannot see the text that falls beyond the limit.
This constraint is a fundamental aspect of how these models operate. If you are working with long documents or extended conversations, you must manage the context window carefully. Exceeding it means losing information, which can lead to less accurate or coherent responses. Efficiently packing your input ensures the model has the necessary context to generate high-quality results.
Why token counts affect API pricing
Commercial LLM APIs typically charge based on the number of tokens processed. This pricing model counts input tokens (the prompt) and output tokens (the generated response) separately. Consequently, the same request can cost a different amount depending on how verbose your prompt is and how long the model’s response turns out to be.
This direct correlation means that more efficient tokenization of a given amount of text translates directly into lower cost. If you can achieve the same result with fewer tokens, you pay less. This is why understanding token counts is essential for anyone building applications that scale. It allows you to optimize your prompts and manage budgets effectively without sacrificing performance. Our model pricing pages show input and output rates per million tokens for each model, so you can see this cost difference directly.
FAQ
What is a token in AI?
A token is the fundamental unit of text that an AI model processes. It can be a word, part of a word, a character, or punctuation, determined by the tokenizer’s specific rules.
Do tokens equal words?
No. Tokens are not the same as words. A single word can be one token or multiple tokens depending on its length and frequency in the model’s vocabulary.
How does tokenization affect cost?
APIs charge per token processed. Since input and output tokens are counted separately, verbose prompts and long responses increase the total cost.