Networks can now read a sentence and remember it. But reading is only half of most language tasks. Translation, the problem that drove this whole field, needs both halves: take in a sentence in English, put out a sentence in French, where the output has different words, different grammar, even a different number of words.
The dream is old. In 1947, before "artificial intelligence" was even a phrase, the mathematician Warren Weaver wrote:
"When I look at an article in Russian, I say: 'This is really written in English, but it has been coded in some strange symbols. I will now proceed to decode.'"
Translation as code-breaking: the meaning is in there, just wrapped in unfamiliar symbols. Machine translation systems eventually took that framing almost literally, and the architecture they arrived at has code-breaking built into its name: encoder–decoder, also called seq2seq (sequence to sequence).
Two networks, one handoff
The design uses two recurrent networks working in relay:
The encoder reads. It is an LSTM from the last part, consuming the English sentence one word at a time, updating its memory at each step, and producing no output at all. Its only job is to understand. When the last word has been read, the encoder's final memory, one fixed-size list of numbers, is taken to be a summary of the entire sentence. This summary is called the context vector.
The decoder writes. A second LSTM starts with the context vector as its opening memory and generates the French sentence one word at a time. And it works in a loop you have seen before if you read ML Basics Part 11: produce a word, feed that word back in as the next input, produce another, and repeat until it emits a special "I'm finished" token. Because it consumes its own output as it goes, this kind of generation is called autoregressive, and it is exactly how ChatGPT produces text today.
(One housekeeping note: neither network sees letters. As covered at the end of ML Basics, words enter as lists of numbers called embeddings, learned so that similar words get similar numbers. Part 7 of this series opens that box properly.)
Training with stabilisers: teacher forcing
Training this pair holds a small trap. Suppose the decoder is learning to produce "J'aime les chats" and its first attempt at word one comes out as "Bonjour", wildly wrong. If we feed that back in as the next input, the rest of the lesson is garbage: the network is now trying to learn word two of a sentence that never existed. Early in training, when almost every guess is wrong, it would never string together enough correct context to learn anything.
The standard fix is called teacher forcing: during training, regardless of what the decoder actually guessed, feed it the correct previous word from the training example. Guess wrong, get told the right answer, continue from there. Like a piano teacher who corrects each wrong note immediately so the student still experiences the rest of the piece properly, it makes learning dramatically faster and more stable.
It has a known cost, with a name worth recognising: exposure bias. The network trains with stabilisers on, always continuing from a correct history, but at test time it must ride without them, continuing from its own possibly-wrong words, a situation it never practised recovering from. One wobbly word can compound into a derailed sentence. Researchers developed ways to wean models off the teacher gradually, but the training-versus-reality mismatch remains a classic theme in this field.
The straw in the middle
Now for the flaw that motivates everything after this. Look again at the diagram: every drop of meaning must pass through the context vector. One fixed-size box, the same size for a three-word greeting and a ninety-word legal clause. The encoder must cram everything into it, and inevitably, as sentences grow, things fall out. Worse, they fall out unfairly: the encoder read left to right with fading memory, so it is the early words that get crowded out first.
Try it. Keep extending the sentence below, then answer the question at the bottom.
The sentence, as the context vector "remembers" it (faded = crowded out of the summary):
With a short sentence, the summary holds everything and translation works beautifully, encoder-decoder systems were a genuine breakthrough, and Google Translate ran on exactly this design in the mid-2010s. But benchmark scores told the same story as the demo: quality fell off a cliff as sentences got longer. The whole architecture was drinking a novel through a straw.
What's next
The fix was not a bigger box. It was a question: why should the decoder rely on a single squeezed summary when the encoder's word-by-word notes still exist? Let the decoder look back at all of them, and pay attention to the ones that matter for the word it is writing right now. That idea is called attention, it is the single most important idea in modern AI, and it is the subject of the next part.