The Transformer: Attention Is All You Need

By 2017, attention was bolted onto every serious translation system, and an awkward truth was becoming visible: the attention bolt-on was doing the interesting work, while the RNN underneath still crawled through text one word at a time, still fading, still slow to train. That year, a team of Google researchers published a paper whose title was a thesis, a provocation, and a spoiler all at once: "Attention Is All You Need."

Delete the recurrence. No loop, no notepad, no conveyor belt. Keep attention, fix the two problems that deleting the loop creates, and stack the result deep. They called the architecture the transformer, and it is, essentially unchanged, the machine inside ChatGPT, Claude, and Gemini today.

Everything, everywhere, all at once

An RNN must finish word 6 before it can start word 7, because word 7 needs the notepad. Remove the notepad and the queue disappears: every word can be processed at the same time, because attention lets any word reach any other word directly, no relay required.

This mattered as much for engineering as for quality. Modern computer chips (GPUs) are magnificent at doing thousands of small calculations simultaneously and mediocre at long chains of one-after-another steps. RNNs are the second thing; transformers are the first. The moment the architecture went parallel, models could be trained on vastly more text in the same time, and the "large" in large language model became possible.

But deleting the loop breaks two things, and the transformer's real content is the two fixes.

Fix one: the sentence must read itself

In the last part, attention connected two different sequences: French words looking back at English words. With the RNN gone, the transformer applies the same trick within a single sentence: every word attends to every other word to work out its own meaning. This is called self-attention.

Why is that needed? Because a word alone is ambiguous. Consider:

The animal didn't cross the street because it was too tired.

What is "it"? You know instantly: the animal (streets don't get tired). The word "it", by itself, contains none of that, its meaning has to be assembled by looking at the rest of the sentence. Self-attention is the assembly mechanism: "it" scores every other word for relevance, finds "animal", and pulls that meaning in. After a round of self-attention, the numbers representing "it" have literally absorbed animal-ness. The same goes for subtler cases: "bank" ends up with different numbers in "river bank" than in "savings bank", because its neighbours voted differently. Vectors that know their context, this is what researchers mean by contextual embeddings.

Queries, keys, and values: three hats per word

How does a word "score every other word"? The transformer gives each word three roles, produced from the word's numbers by three learned transformations. The names are database jargon, but the idea is a library:

• The query (Q) is the question this word is asking: what you type into the library's search box. For "it": "seeking: a nearby thing that could be doing something."
• The key (K) is the label this word shows to searchers: the card in the catalogue. For "animal": "on offer: a living creature, main character of this sentence."
• The value (V) is what the word actually hands over if selected: the book itself, the usable content of the word.

Scoring is then the familiar similarity trick from last part: compare this word's query against every word's key (dot product), softmax the scores into shares, and blend all the values according to those shares. Each word does this simultaneously, every word searching the whole sentence, every word being searched. One important consequence of splitting the roles: matching and content are separate. A word can be easy to find without that changing what it delivers, the same way a good catalogue entry doesn't rewrite the book.

For the curious: the full recipe per word is softmax(Q·Kᵀ / √d) · V. The one piece not covered before is the ÷ √d: with long vectors the dot products come out large, which pushes softmax into winner-takes-all extremes; dividing by the square root of the vector length keeps the scores in a range where learning stays stable. The Q, K, and V transformations are just weight matrices, knobs, found by gradient descent like every other number in this story. A miniature worked example: if q_it = [2, 0, 1], k_animal = [1.5, 0, 1] and k_street = [0, 1, 0.2], then the scores are q·k_animal = 4.0 and q·k_street = 0.2 — "it" matches "animal" twenty-to-one before softmax even sharpens the gap.

Fix two: several searchlights are better than one

One round of self-attention gives each word one blended glance at the sentence, but words relate to each other in several ways at once. In our example, "it" wants its referent ("animal"), but it might also care about its grammatical role (subject of "was"), and about its immediate neighbours. One searchlight cannot point three directions.

So the transformer runs several attentions in parallel, typically 8 or more, each with its own separately-learned Q, K, and V transformations. Each one is called a head, and during training each head drifts toward its own speciality: one ends up tracking who-refers-to-what, another verb-object pairs, another simple adjacency. Think of a committee of readers, each with different glasses, whose reports are stitched together at the end. This is multi-head attention, and it is why the same sentence can be simultaneously understood grammatically, semantically, and positionally.

Try it: watch a sentence read itself

Click any word below and see where its attention goes, the darker the highlight, the more that word contributes to the clicked word's updated meaning. Then switch heads: the same sentence, the same click, a completely different pattern of glances. (Weights hand-tuned for illustration; a real model learns them.)

The missing ingredient: word order

One casualty of deleting the reading loop: an RNN knew word order for free, it experienced the sentence in sequence. Self-attention does not; it treats the input as a bag of words that all look at each other. Left uncorrected, "dog bites man" and "man bites dog" would be identical, which is a problem for exactly one of those parties.

The fix is positional encoding: before any attention happens, each word's numbers get a position stamp added, a distinctive wavy pattern of values that means "I am word 1", "I am word 2", and so on. (The original design built the stamps from sine and cosine waves at many frequencies, which has the nice property of working for sentences of any length.) With positions stamped in, a head that cares about order, like the neighbours head in the demo, can learn to use it, and the two dog-and-man sentences become as different as they deserve to be.

Stacking it up

One round of self-attention plus one small feedforward network, the very architecture from Part 1, applied to each word to digest what attention gathered, makes one transformer block. (Each block also includes some numerical plumbing, residual connections and normalisation, whose job is simply to keep signals stable as things get deep.)

Then you stack: the original paper used 6 blocks, modern LLMs use dozens to over a hundred. Each block re-reads the output of the one below, so the understanding compounds: early blocks resolve local grammar, middle blocks assemble phrases and references, deep blocks handle long-range structure and meaning, the same "layers of increasing abstraction" story you may remember from the neurons part of ML Basics, now applied to language.

What's next

The 2017 transformer was actually two towers, an encoder that reads and a decoder that writes, joined by attention, direct descendants of the relay from Part 3. But the field quickly discovered that you rarely need both. Keep only the writing tower and you get GPT. Keep only the reading tower and you get BERT. What each half is for, and how a decoder-only model manages the two-phase trick of digesting your prompt and then answering it, is the subject of the next part.