The original transformer from the last part was built for translation, so it had two towers: an encoder to read the English, a decoder to write the French, with attention bridging them. Elegant, but most language tasks are not translation, and the field quickly noticed that for many jobs, one tower is all you need. Which tower you keep determines what kind of intelligence you get.
The writer: decoder-only, a.k.a. GPT
Keep only the writing tower and you get a decoder-only model. That is what the D in the middle of GPT belongs to: Generative Pre-trained Transformer. ChatGPT, Claude, Gemini, Llama, essentially every chatbot you have used is this flavour.
A decoder-only model has one defining rule, inherited from its job. It is trained to predict the next token (the game from ML Basics Part 11), and you cannot learn to predict the future if you are allowed to peek at it. So its self-attention wears blinkers: each word may only attend to the words before it, never after. This one-way rule is called a causal mask (or masked self-attention). During training, that means every position in every sentence is simultaneously a fair little prediction exercise. During generation, it matches reality anyway, the future tokens don't exist yet.
When you send a chatbot a message, it works in two phases, and the difference explains something you have probably felt:
Phase 1, reading your prompt. All the prompt's tokens are known upfront, so the model processes them in one parallel pass, the transformer's great trick, with the causal mask still applied within it. The output of this phase is nothing visible: just context-aware numbers for the whole prompt, the model's digested understanding.
Phase 2, writing the reply. Now the future genuinely doesn't exist yet, so parallelism is off the table: the model predicts one token, appends it, and runs again, each new token attending back over the prompt and everything written so far. Strictly one at a time. This is why a chatbot ingests your five-paragraph question near-instantly, then types its answer out word by word: reading is parallel, writing is sequential.
The reader: encoder-only, a.k.a. BERT
Keep only the reading tower instead and you get an encoder-only model, the most famous being BERT (2018), which quietly powered Google Search for years. No mask, no blinkers: every word attends in both directions, left and right. Fittingly, the B in BERT stands for bidirectional.
Why does that matter? Because for understanding (rather than generating), the future of the sentence is right there, and it often holds the answer:
The bank was slippery after the rain.
At the word "bank", a left-only reader knows almost nothing, "The bank was..." could go either way. The disambiguating evidence ("slippery", "rain") comes later. A bidirectional model reads the whole sentence at once and settles "bank" = riverbank using words on both sides. The price: a model with no mask has seen everything, so there is nothing for it to predict next, it cannot generate text. Its output is understanding: context-aware numbers for each word, which you then use for search ranking, classifying a review as positive or negative, spotting names and dates in contracts, routing support tickets.
The translator: both towers
The full encoder–decoder transformer survives too, best for tasks that transform one text into another: translation, summarisation, "rewrite this politely". The encoder reads the source bidirectionally (understanding needs both directions), the decoder writes the result causally (writing always faces forward), and cross-attention lets every written word consult the fully-understood input. It is the Part 3 relay, rebuilt with Part 5 machinery, and no straw in the middle.
| Flavour | Attention | Superpower | Examples |
|---|---|---|---|
| Decoder-only | One direction (causal) | Generating text | GPT, Claude, Gemini, Llama |
| Encoder-only | Both directions | Understanding text | BERT, RoBERTa |
| Encoder–decoder | Both, then causal | Transforming text | T5, translation systems |
Try it: the mask
Click any word below, then flip between the two modes. The highlight shows which words the clicked word is allowed to consult when working out its own meaning.
Click "bank" and flip modes a few times, that single flip is the whole strategic difference between the model behind ChatGPT and the model behind a search engine. One gave up seeing the future in exchange for the power to create it; the other gave up creating in exchange for seeing everything.
What's next
You now know the architecture end to end. One question remains, and it is the one nontechnical audiences ask most often: when a company "ships a model", what actually is it? A database? A search index? A very long list of if-then rules? The final part opens the box, counts what's inside, and lets you weigh a few famous models on an interactive scale.