AI glossary
Attention Mechanism
An attention mechanism is a component in a neural network that lets the model weigh the relevance of different parts of the input when producing each part of the output. It computes a set of weights that determine how much focus to place on each input element, allowing the model to selectively attend to the most important information for the current task.
This mechanism solves a fundamental problem in sequence processing: knowing which parts of a long input matter most for a specific output. Instead of treating all input data equally, the model learns to assign higher weights to relevant elements and lower weights to noise or irrelevant context. This dynamic weighting allows for more accurate and context-aware predictions, particularly in tasks like translation, summarization, and language understanding.
How self-attention works
In the self-attention used by transformers, each element of the input sequence is compared against every other element, including itself, to compute attention weights. This comparison is done by projecting each element into three vectors: a query, a key, and a value. The query represents what a given position is looking for; the keys represent what content is available at every position; the values contain the actual information at each position.
The model computes a relevance score between each query and every key, commonly using a dot product that is then scaled. These scores are converted into weights that sum to one using the Softmax function. The output for that position is produced as a weighted sum of all the value vectors, using those weights.
Because every position can directly attend to every other position in a single step, self-attention lets the model relate any two elements in a sequence regardless of how far apart they are. This is a significant improvement over earlier architectures that had to pass information step by step through intermediate positions, which often diluted the signal for distant relationships.
Multi-head attention
Instead of computing a single attention pattern, transformers typically use multi-head attention. This involves several attention computations, or “heads,” running in parallel, each with its own learned query, key, and value projections. This parallel structure allows the model to capture different types of relationships simultaneously.
Different heads can learn to focus on different types of relationships within the same input. For example, one head might attend to nearby words to capture local grammar rules, while another head focuses on distant words to understand broader context or long-range dependencies. The outputs of all heads are combined to form the final result, providing a rich, multi-faceted representation of the input data. This approach is a core component of the Transformer architecture, which relies on these diverse attention patterns to process Tokens effectively.
Origin
Attention mechanisms were first used in an earlier form for neural machine translation, where they helped align source and target languages by highlighting relevant words. The mechanism was later formalized as “scaled dot-product attention,” providing an efficient, mathematically precise way to compute these weights.
The full transformer architecture built entirely around self-attention was introduced in the 2017 paper “Attention Is All You Need” by Ashish Vaswani and colleagues at Google. This paper demonstrated that self-attention could replace recurrent and convolutional layers entirely, leading to faster training and better performance on various tasks. The introduction of this architecture marked a pivotal moment in natural language processing, enabling the development of large language models that rely on embedding layers to convert text into numerical representations.
Why it replaced recurrent processing
Earlier architectures, such as recurrent neural networks, processed a sequence one step at a time. They passed information forward through a hidden state, which made it hard to directly relate distant elements and limited how well the computation could be parallelized during training. This sequential nature created a bottleneck, especially for long sequences, as the model had to process each token in order before moving to the next.
Self-attention computes relationships between all positions in the sequence at once. This allows much greater parallelization on modern hardware, such as GPUs and TPUs, making it practical to train much larger models on much larger amounts of data. The ability to process all tokens simultaneously also helps the model retain long-term dependencies without the vanishing gradient problems that often plagued earlier recurrent approaches. Additionally, the use of KV cache in inference further optimizes the efficiency of attention mechanisms by storing previously computed key and value pairs, reducing redundant calculations during text generation.
FAQ
What is attention in neural networks?
Attention is a technique that allows a model to dynamically weigh the importance of different input elements when generating an output. It enables the network to focus on relevant parts of the data, improving performance in tasks like translation and image recognition.
How does self-attention work?
Self-attention compares each element in a sequence with every other element to compute relevance scores. It projects inputs into query, key, and value vectors, then uses these to calculate weights that determine how much information from each position contributes to the final output.
What is the difference between self-attention and multi-head attention?
Self-attention computes a single attention pattern for the input, while multi-head attention runs multiple attention layers in parallel. Each head learns different relationships, allowing the model to capture diverse features and contexts simultaneously before combining the results.
Why are transformers better than RNNs for long sequences?
Transformers use self-attention to process all tokens in parallel, avoiding the sequential bottlenecks of RNNs. This allows them to capture long-range dependencies more effectively and train faster on modern hardware, making them ideal for large-scale language models.