Step by Step
Q
Query — what am I looking for?
For every word being processed, attention computes a Query representing what that word is currently trying to find or relate to among the other words in the sequence.
Example: while processing the word "it," the Query represents the model's attempt to figure out what "it" actually refers to.
K
Key — what does each word offer?
Every word in the sequence also produces a Key, representing a summary of its content, which gets compared against Queries to determine relevance.
Example: the word "trophy" produces a Key summarizing its content, which can be compared against the Query from "it" to check relevance.
V
Value — the actual content retrieved
Once relevance scores are computed by comparing Queries and Keys, those scores weight each word's Value, and the weighted combination of Values becomes the model's actual retrieved information — the highlighted, relevant content.
Example: if "trophy" scores highly relevant to "it," then "trophy"'s Value contributes heavily to the final representation computed for "it."
Applied Walkthrough
1
A Transformer processes the sentence "The trophy didn't fit in the suitcase because it was too big," and needs to determine what "it" refers to.
2
The Query for "it" is compared against the Keys of both "trophy" and "suitcase," computing relevance scores for each.
3
Given the context (something being "too big" to fit), the attention mechanism correctly assigns a higher relevance score to "trophy" than to "suitcase."
4
This means "trophy"'s Value contributes more heavily to the final representation of "it," correctly resolving this classic ambiguous pronoun reference — exactly the kind of task attention is specifically good at.
Exam Application
Exams test whether you can define Query, Key, and Value and describe how they work together (Query compared against Keys to compute relevance, then Values combined based on those relevance scores). The classic "trophy/suitcase" pronoun resolution example is a frequently used illustration of why attention matters.
⚠ Common Trap
The most common trap is confusing what each of the three components (Query, Key, Value) actually represents. A helpful way to remember: Query is what you're searching FOR, Key is what each candidate word OFFERS for comparison, and Value is the actual CONTENT you retrieve once relevance is determined.
✓ Quick Self-Check
1. What does the Query represent in the attention mechanism?
What the current word is looking for or trying to relate to among other words.
Tap to reveal / hide
2. What does the Key represent in the attention mechanism?
A summary of what a given word offers, compared against Queries to determine relevance.
Tap to reveal / hide
3. What does the Value represent in the attention mechanism?
The actual content retrieved, weighted by the computed relevance scores.
Tap to reveal / hide
4. In the sentence "The trophy didn't fit in the suitcase because it was too big," what should "it" correctly attend to?
"Trophy," since the context (something being too big to fit) points to the trophy rather than the suitcase.
Tap to reveal / hide
5. What is a helpful way to remember what Query, Key, and Value each represent?
Query is what you're searching for; Key is what each candidate offers for comparison; Value is the actual content retrieved.
Tap to reveal / hide