AI glossary
Temperature (LLM)
Temperature is a parameter that controls the randomness of a language model’s output by scaling the raw prediction scores (logits) before they are converted into a probability distribution over possible next tokens, typically using the softmax function.
When you ask a large language model to generate text, it doesn’t just pick the next word at random. It calculates a score for every word in its vocabulary. The temperature setting acts as a dial that sharpens or flattens these scores. A low temperature makes the model pick the most likely word every time, while a high temperature encourages it to consider less obvious choices, leading to more creative or varied text.
What is temperature in LLM?
At its core, the temperature parameter ai adjusts how confident the model appears in its predictions. Before the model outputs a token, it assigns a raw score, or logit, to every possible next token. These logits are divided by the temperature value. This division changes the spread of the scores. If you divide by a number less than one, the differences between scores grow larger. If you divide by a number greater than one, the differences shrink.
After this scaling, the scores are passed through the softmax function. Softmax converts these scaled scores into probabilities that add up to one. The model then samples from this final probability distribution to decide which token to output next. This process happens for every word in the generated sequence. By adjusting the temperature, you directly influence the shape of that distribution and, consequently, the style and consistency of the generated text.
How temperature works
The mechanism relies on the relationship between the raw logits and the final probability distribution. At each generation step, the model computes a score for every possible next token in its vocabulary. These raw scores reflect how well each token fits the context provided by the previous tokens.
Before sampling the next token, these logits are divided by the temperature value. This step is crucial because it changes the relative differences between the scores. Dividing by a temperature below one makes the differences between logits larger before the softmax step. This results in a sharper, more peaked probability distribution. The highest-scoring token becomes much more likely than the others. Conversely, dividing by a temperature above one shrinks those differences. This produces a flatter, more even distribution where many tokens have similar probabilities.
The softmax function then takes these adjusted scores and normalizes them. The output is a set of probabilities that the model uses to select the next word. The temperature parameter effectively controls the entropy of this distribution. Lower entropy means the model is more deterministic. Higher entropy means the model is more exploratory.
Low vs high temperature
Understanding the difference between low and high temperature settings helps you choose the right tool for your task. The behavior changes significantly depending on whether the value is below or above one.
A lower temperature, typically closer to zero, makes the model’s output distribution sharper. This means the model more consistently selects the highest-probability token. The result is output that is more focused, deterministic, and often repetitive. If you set the temperature to zero, or a value very close to it, you approximate greedy decoding. In this mode, the model always picks the single most likely next token at every step. This is useful when you need precision and consistency.
On the other hand, a higher temperature flattens the distribution. This gives lower-probability tokens a greater chance of being selected. The model becomes less certain and more willing to take risks. This produces output that is more varied and creative. However, this increased randomness can lead to less coherent or accurate text. The model might generate plausible-sounding but factually incorrect statements.
A temperature of one leaves the model’s natural output distribution unscaled. This is the default behavior in many systems. It represents the model’s inherent confidence without artificial amplification or dampening. Developers typically experiment with several values within an API’s allowed range to find a balance between creativity and accuracy for their specific task.
Temperature vs top-p sampling
Temperature is one of several sampling parameters commonly exposed in LLM APIs. It is often used alongside top-p (nucleus) sampling. While both affect randomness, they work in different ways.
Top-p sampling restricts the model to sampling only from the smallest set of tokens whose cumulative probability exceeds a chosen threshold. This cuts off the long tail of very unlikely tokens regardless of how flat or sharp the overall distribution is. For example, if p is set to 0.9, the model will only consider tokens that make up the top 90% of the probability mass.
The two parameters address different aspects of randomness. Temperature reshapes how peaked or flat the whole probability distribution is. It changes the relative likelihood of all tokens. Top-p restricts which tokens are even eligible to be sampled. It ignores the tail of the distribution entirely. APIs commonly allow adjusting both together. You can use top-p to remove extreme outliers and temperature to fine-tune the sharpness of the remaining options.
Choosing a temperature
Selecting the right temperature depends on your specific use case. There is no single correct value. You must match the setting to the desired outcome.
Lower temperatures are generally preferred for tasks that need consistent, predictable, and factually precise output. This includes code generation, where syntax errors are unacceptable, or data extraction, where you need exact values. It is also ideal for answering questions with a single correct answer. In these scenarios, you want the model to stick to the most probable path.
Higher temperatures are generally preferred for tasks that benefit from variety and creativity. This includes brainstorming sessions, creative writing, or generating multiple different options to choose from. If you are writing a marketing email or a poem, you want the model to explore different phrasing and ideas. A higher temperature allows for that exploration.
Many developers use a middle-ground approach. They might start with a moderate temperature and adjust based on the results. If the output is too robotic, they increase the temperature. If it is too random or hallucinatory, they lower it. This iterative process is a key part of prompt engineering. It ensures the model behaves as expected for your specific application.
FAQ
What does temperature do in an LLM?
Temperature scales the raw prediction scores before they are converted into probabilities. It controls the randomness of the output by making the probability distribution sharper or flatter.
What is the difference between temperature and top-p?
Temperature changes the shape of the probability distribution, while top-p restricts the pool of eligible tokens. Temperature affects how likely lower-probability tokens are, whereas top-p removes tokens below a cumulative probability threshold.
What temperature should I use for code generation?
You should use a low temperature, close to the minimum your API allows. This ensures the model selects the most likely tokens, resulting in more consistent and syntactically correct code.
Is a higher temperature better for creative writing?
Yes, a higher temperature is generally better for creative writing. It encourages the model to choose less obvious words, leading to more varied and imaginative text.