AI glossary
Cross-Entropy Loss
Cross-entropy loss measures the difference between two probability distributions: the true labels and the model’s predicted probabilities. It is the standard loss function for classification tasks and for training language models. The value is lower when predictions align with reality and approaches zero when the model assigns a probability of 1 to the correct class.
Consider a simple scenario: a model predicts a 99% chance that an image contains a cat, but the image actually shows a dog. The penalty for this error depends on how “wrong” the model was. Cross-entropy captures this uncertainty precisely, rewarding confidence when correct and punishing it heavily when wrong. This is why it is the default choice for training classifiers.
Cross-entropy formula
The mathematical definition varies slightly depending on whether you are dealing with multiple classes or a single binary outcome. For multi-class problems, the categorical cross-entropy loss is defined as:
L = - sum over classes i of y_i × log(p_i)
Here, y represents the true distribution (often one-hot encoded) and p represents the predicted probability distribution. If the label is one-hot (meaning only one class is correct and its value is 1, while others are 0), the formula simplifies significantly. The loss reduces to:
L = -log(p_correct)
This simplified form highlights why the metric is effective: it focuses entirely on the probability assigned to the correct class. During training, this loss is calculated for every example in a batch and then averaged to update the model weights.
The choice of logarithm base determines the unit of measurement. Using the natural logarithm (ln) results in units of nats. Using log base 2 results in units of bits, which connects the concept directly to information theory.
Worked example
To understand how the math translates to actual training behavior, let’s look at a concrete calculation. Suppose a classifier outputs raw logits of [2.0, 1.0, 0.1]. Applying the softmax function converts these logits into a probability distribution of [0.659, 0.242, 0.099].
If the correct class is the first one, the loss is calculated using the natural logarithm: loss = -ln(0.659) ≈ 0.417
If the correct class is actually the third one (a mistake by the model), the loss is: loss = -ln(0.099) ≈ 2.31
The penalty increases dramatically when the model is confidently wrong. If the correct class receives a probability of only 0.01, the loss jumps to approximately 4.61. As that probability approaches zero, the loss grows without limit. This unbounded nature ensures that the model is aggressively penalized for high-confidence errors, driving it to correct its mistakes faster.
Binary cross-entropy (log loss)
When dealing with binary classification or multi-label problems, we use binary cross-entropy, also known as log loss. This variant calculates the error for a single probability p of the positive class against a label y (which is either 0 or 1).
The formula is: L = -[y × log(p) + (1 - y) × log(1 - p)]
This function is typically paired with a sigmoid activation function. Binary cross-entropy is the usual loss function when several independent yes/no decisions are made at once, such as tagging an image with multiple relevant labels. Unlike categorical cross-entropy, which assumes mutually exclusive classes, binary cross-entropy allows for overlapping labels.
Why cross-entropy instead of mean squared error
A common question is why we don’t just use mean squared error (MSE) for classification. While MSE works, it suffers from a critical flaw when used with activation functions like softmax or sigmoid.
With cross-entropy, the gradient is large when the model is confidently wrong. This means the model learns quickly from its worst mistakes. In contrast, MSE produces weak gradients in these situations, causing the learning process to stall.
Furthermore, the gradient of cross-entropy with respect to the logits, after applying softmax, simplifies neatly to (p - y). This simplicity keeps training numerically stable and efficient. The direct relationship between the error signal and the prediction error makes cross-entropy the preferred choice for most classification tasks.
Cross-entropy, entropy and KL divergence
Cross-entropy is deeply rooted in information theory. It relates to the entropy of the true distribution and the Kullback-Leibler (KL) divergence between the true distribution p and the predicted distribution q.
The relationship is expressed as: H(p, q) = H(p) + KL(p || q)
Here, H(p) is the entropy of the true distribution, and KL(p || q) is the KL divergence. Since the true labels are fixed during training, minimizing cross-entropy is mathematically equivalent to minimizing the KL divergence. KL divergence measures how much extra information is needed when q is used in place of p, so lowering cross-entropy brings the predictions closer to the true labels.
Cross-entropy in language models
In the context of large language model, LLM training, cross-entropy is the primary objective. Models are trained on next-token prediction. At each position in the sequence, the loss is calculated as the negative log of the probability the model assigned to the actual next token.
This process allows the model to learn the statistical structure of language. A common metric derived from this loss is perplexity. Perplexity is calculated as exp(average cross-entropy per token) when the natural log is used. Lower perplexity indicates a better model, as it predicts the next token with higher probability.
Cross-entropy in PyTorch and Keras
Implementing cross-entropy is straightforward in modern frameworks, but subtle differences exist.
In PyTorch, nn.CrossEntropyLoss expects raw logits and applies log-softmax internally. This is efficient and numerically stable. For binary tasks, nn.BCEWithLogitsLoss is used.
import torch
import torch.nn as nn
logits = torch.tensor([[2.0, 1.0, 0.1]]) # raw model outputs, no softmax
target = torch.tensor([0]) # index of the correct class
loss = nn.CrossEntropyLoss()(logits, target)
print(loss.item()) # about 0.417In Keras, you choose between categorical_crossentropy for one-hot labels and sparse_categorical_crossentropy for integer labels. Both accept from_logits=True if your model outputs raw logits, avoiding the need for a separate softmax layer.
FAQ
What is the difference between cross-entropy and log loss?
Log loss is another name for cross-entropy. In practice the term most often refers to binary cross-entropy, the version used for yes/no outcomes.
Why is cross-entropy preferred over mean squared error?
Cross-entropy provides larger gradients when a model is confidently wrong, allowing for faster learning. MSE gradients can vanish with sigmoid/softmax activations, slowing down training.
What is label smoothing?
Label smoothing replaces the hard one-hot target with a softened distribution (e.g., 0.9 on the correct class). This reduces overconfidence and can improve generalization.
How do I choose between PyTorch’s CrossEntropyLoss and BCEWithLogitsLoss?
Use CrossEntropyLoss for multi-class problems with integer targets. Use BCEWithLogitsLoss for binary classification or multi-label problems where outputs are independent probabilities.