Module 9 showed what attention does — the spotlight. Now we open the hood: how does each word actually compute those attention scores? The answer is a beautifully simple search mechanism built from three vectors: Query, Key, and Value.
The mechanism
Every word makes three versions of itself
Here's the trick. For each word (token), the model builds three different vectors from it — each answering a different question. These are learned transformations, just like every other weight in the network.
Q
Query
"What am I looking for?"
K
Key
"What do I have to offer?"
V
Value
"What information do I pass on?"
Layman example — searching on YouTube
You type a search — that's your Query ("what I'm looking for"). Every video has a title/tags — that's its Key ("what I offer"). YouTube matches your query against every key to score relevance. Then it gives you the actual video content — that's the Value ("what I pass on"). Attention is exactly this: a word queries, other words advertise keys, and the best matches hand over their values.
The flow
Query · Key → score, then weighted sum of Values
The whole attention calculation is three steps, and you already understand each piece:
# 1. score how well this word's Query matches each other word's Key
score = Query · Key(a dot product = how aligned they are)
# 2. turn scores into weights that sum to 1
weights = softmax( all scores ) (remember softmax from Module 3!)
# 3. the new, context-rich version of the word:
output = Σ ( weight × Value ) (a weighted blend of the values)
What this achieves
The word ends up enriched with the meaning of the words it attended to. "bank" that queried and matched "river" absorbs some of "river"'s value → it now means river-bank. This is how a plain word-vector becomes context-aware. (Note: in a text-generating model this is causal — a word can only query words before it, as we saw in Module 9.)
The full pipeline — press Run to watch it flow
One word's vector → splits into Q/K/V → scores → softmax → enriched output.
🎮 Game 1 — the mechanism
Run the Q·K·V search yourself
The word "learns" sends out its query. Each earlier word advertises a key. Press Match to score them, then Collect values to see "learns" absorb the winner's meaning.
"The robot learns fast" — from your notes
Query from "learns" · Keys from earlier words · Values collected.
🔍 Query — from "learns"
"Are you a noun that is performing me?"
The big picture
The attention heatmap — every word querying every word
Game 1 showed one word querying. But every word does this at once. The standard way to visualize all of it is a heatmap grid: rows are the queries (the word looking), columns are the keys (the words being looked at). Each cell's brightness = how much that row-word attends to that column-word.
"The cat sat because it was tired"
Hover a row to highlight it. Brighter = stronger attention.
In writing mode, the upper-right triangle is masked 🔒 — words can't attend to the future.
Look at the "it" row — its brightest cell is "cat". That's the model resolving what "it" refers to, shown as a single glowing square. This grid is what people mean by "an attention pattern."
🎮 Game 2
The four steps, laid out
Your notes tie it together with "The robot learns fast." Step through the exact Query → Key → Alignment → Value flow.
How "learns" connects to "robot"
One step at a time.
Intuition
What questions do queries ask?
The query vectors are learned, so we can't read them literally — but we can imagine the kinds of "questions" a word asks about earlier words. Your notes list several:
🎮 Game 3
Pick a question, see which word answers
Choose a "question" a word might ask, and watch which earlier word lights up as the best match. Each is a different alignment the model can learn.
Different queries find different matches
Sentence: "The hungry brown dog quickly ate"
👆 pick a question above
Scaling up
Multi-Headed Attention: a team of specialists
One attention calculation finds one kind of relationship. But sentences have many at once — grammar, adjectives, adverbs, references. So the model runs several attention "heads" in parallel, each learning to spot a different kind of connection, then combines them.
Layman example — friends on a jigsaw puzzle (from your notes)
Imagine friends assembling a jigsaw. Instead of one person doing everything, each takes a job — one finds edge pieces, one does the sky, one does a specific pattern. They work in parallel, then periodically combine their progress for a better picture, then split again. Multi-head attention is exactly this: each head is a friend hunting for one type of relationship.
🎮 Game 4
Watch the heads find different things
From your notes: "The heavy robot learned fast." Toggle each head on/off and watch the different connections it discovers. Together, they build a deep, multi-dimensional understanding.
Three heads, three kinds of connection
Click a head to toggle its links on the sentence.
Each head learns its specialty during training. Their outputs are concatenated (joined together) and passed on — giving the model a far richer understanding than any single head could.
🎯 Check your understanding
Quick challenge
1. What does the Query vector represent?
2. How is an attention score computed?
3. What becomes the word's new context-rich representation?
4. Why use multiple attention heads?
Takeaway
What you learned
Attention computes its scores with three vectors per word: Query ("what I'm looking for"), Key ("what I offer"), Value ("what I pass on") — like a search engine. A word's Query · Key gives an attention score, softmax turns scores into weights, and the output is a weighted sum of Values — so the word absorbs the meaning of what it attended to. Multi-head attention runs many of these in parallel (jigsaw friends), each finding a different relationship, then combines them.
🎉 That completes Part C — you now understand attention inside and out. Next → Part D, Module 11: The Transformer + Encoder vs Decoder. We'll assemble everything into the full architecture and meet the two great families: the "Understander" (encoder) and the "Writer" (decoder) — finally naming GPT vs BERT.