Attention Mechanism
An attention mechanism lets a neural network weigh how relevant each part of its input is to every other part when producing an output.
What is Attention Mechanism?
In a sentence such as "the robot dropped the box because it was heavy", attention helps the model link "it" to "the box". Self-attention, where every token attends to every other token, is the core of transformers. Its cost grows rapidly with input length, which limits context windows.
How it works
Each token produces query, key and value vectors; similarity between queries and keys sets weights used to combine the values.
← All Artificial Intelligence & Superintelligence definitions · ← Back to the A–Z definitions