The last part ended with a design flaw: the decoder writes an entire translation while looking at nothing but one fixed-size summary of the input. Everything the encoder understood, word by word, was thrown away except the final squeeze.

But think about how a human translator works through a long sentence. They do not read the English once, close their eyes, and write the French from memory. They keep the English in front of them, and as they write each French word, their eyes flick back to the relevant part of the English, the phrase they are translating right now, while the rest sits in peripheral vision. Different output word, different glance.

That is the entire idea of attention, and the fix for the bottleneck follows immediately: stop discarding the encoder's word-by-word notes. Keep all of them. Then, for each word the decoder writes, let it look back across every input word's note and blend them, giving most weight to whichever inputs matter for this word, right now.

As the researchers who introduced it (Dzmitry Bahdanau and colleagues, in 2014) put it:

"This frees the model from having to encode a whole source sentence into a fixed-length vector, and also lets the model focus only on information relevant to the generation of the next target word."

The straw is replaced by a searchlight.

How the highlighter decides where to shine

The mechanism has three steps, each of them plain arithmetic:

Step 1: score every input word for relevance. The decoder has a note describing what it needs right now ("I am about to write the subject of the sentence"). Each input word has the note the encoder made for it. Relevance is measured by how similar the two notes are, and since notes are lists of numbers, similarity is simple: multiply the lists together, position by position, and add it up. Lists that point the same way give a big total; unrelated lists give a small one. The result is one score per input word.

Step 2: turn scores into shares. Raw scores are awkward, they can be any size, so they are passed through a function called softmax that converts them into percentages: all positive, summing to exactly 100%, with bigger scores getting disproportionately bigger shares. These percentages are the attention weights. A weight of 72% on "go" means: for the word being written right now, "go" gets 72% of the spotlight.

Step 3: blend. Build a custom summary for this one output word by mixing all the input notes according to their weights, 72% of this note, 21% of that one, a dash of the rest. The decoder writes its word using that blend, plus its own state.

Then, and this is the crucial part, the next output word starts the whole process again. New need, new scores, new weights, new blend. The decoder never reuses a stale summary; every word it writes gets a summary custom-made for it. That is what the fixed context vector could never do.

Try the highlighter

The English sentence from last part's demo is below, along with the French words a decoder might write. Pick an output word and watch where the model's highlighter lands, or drag the relevance scores yourself and see how softmax converts them into shares of the spotlight. Notice how sharply the focus moves when you switch output words.

The decoder is currently writing:

Two things to play with. First, switch between output words and watch the highlight jump, writing "rouge" shines the light on "red", writing "garée" moves it to "parked". One input sentence, four completely different summaries. Second, drag any one slider upward and watch every other share shrink: softmax makes the weights compete for a fixed 100%, so attention is always a choice about priorities, not just volume.

For the curious: step 1's "multiply position by position and add up" is the dot product. Step 2 is softmax(s)ᵢ = eˢⁱ / Σⱼ eˢʲ, the same e ≈ 2.718 from ML Basics Part 9; the exponential is what makes high scores dominate. Step 3 is a weighted average: blend = Σᵢ weightᵢ × noteᵢ. In the original attention paper this blend was handed to the decoder LSTM alongside its own hidden state at every output step. Nothing in the three steps has any adjustable parameters of its own, the learning happens in the networks that produce good notes for attention to compare.

Why this was such a big deal

Attention solved the straw problem: translation quality stopped collapsing on long sentences, because the first word of the input is exactly as reachable as the last, one glance away, whenever it is needed. No more relay through a single overloaded box.

But it did something bigger. Notice what attention is: a way for one position in a sequence to reach directly to any other position and pull in what it needs, with the reach decided by learned relevance, not by distance. Memory without decay. No conveyor belt, no gates, no fading notepad, the problems the previous two parts wrestled with simply do not arise for a mechanism that has no notion of "far away".

Which raises a question that sounds almost cheeky. The RNN under all of this still reads one word at a time, still crawls, still forgets, and now attention is doing the real work of connecting words to words. So... do we still need the RNN at all?

What's next

In 2017, a team at Google answered with the most confidently titled paper in the field: "Attention Is All You Need." Throw away the recurrence entirely, keep attention, add a fix for word order, and stack it deep. The result is the transformer, the architecture inside ChatGPT, Claude, and Gemini, and it is the subject of the next part.