The last part gave networks a notepad: a hidden state, carried from word to word, updated by the same rule at every step. That phrase, the same rule at every step, was the elegant part. It is also the flaw.

The tyranny of compound multiplication

Part of the update rule is: take the old note and multiply it by a weight before blending in the new word. One multiplication is harmless. But reading a 100-word paragraph means applying that same multiplication 100 times, and repeatedly multiplying by the same number is a runaway process in one direction or the other:

  • If the weight is even slightly less than 1, old information shrinks exponentially. At 0.9 per step, after 50 words only half a percent of the first word's influence remains; after 100 words, essentially nothing. The network structurally cannot remember the start of a long sentence. This is the vanishing problem.
  • If the weight is even slightly more than 1, the opposite: at 1.1 per step, the first word's influence has grown 117-fold after 50 steps and 13,000-fold after 100. The note gets swamped, and numbers blow up. This is the exploding problem.

There is no comfortable middle. The weight would have to be exactly 1 to carry information faithfully, and a learned weight never lands exactly anywhere.

It gets worse. Training has the same disease in reverse. Recall from ML Basics that a network learns by tracing blame backwards, working out how much each weight contributed to the error. In an RNN, blame for a mistake at word 100 must travel backwards through every step to reach word 1, being multiplied by that same weight at each hop. So the blame signal vanishes (or explodes) exactly the way the memory does, and the network barely learns anything about long-range connections even when they matter most. (The formal name for tracing blame back through the steps is backpropagation through time, which might be the most science-fiction name in all of machine learning.)

Feel the compounding

One slider below sets the per-step carry weight; the other sets how many words the network reads. The curve shows how much of the first word's influence survives at each step.

Try 0.95, gentle fading, and slide the word count up: fine for a short sentence, hopeless for a paragraph. Try 1.05 and watch it leave the chart. The window of workable weights narrows as texts get longer, until it closes entirely.

The fix: a conveyor belt with gates

The solution arrived in 1997, from Sepp Hochreiter and Jürgen Schmidhuber, and it carried language AI for the next two decades: the Long Short-Term Memory cell, or LSTM.

The core insight: the problem is the mandatory multiplication. So the LSTM adds a second memory track, often pictured as a conveyor belt, that runs through the cell almost untouched. Information placed on the belt just rides along, step after step, no repeated shrinking. Alongside the belt, the LSTM keeps the familiar working note for what is relevant right now. Long-term memory and short-term memory, hence the name.

What controls the belt is the clever part: three gates, each a small learned valve that opens between 0 (fully closed) and 1 (fully open):

  • The forget gate decides what to erase from the belt. Open (≈1) means "keep this memory"; closed (≈0) means "wipe it".
  • The input gate decides what from the current word deserves a place on the belt.
  • The output gate decides how much of the belt to consult for this step's working note and prediction.

And crucially, the gates are not fixed. They are tiny neural networks themselves, with weights learned by the same gradient descent as everything else, that look at the current word and the current note and decide, in context, what to keep, add, and use. The network learns when to remember and when to forget.

long-term memory (the conveyor belt) × forget gate erase or keep? + input gate worth adding? ⤓ output gate what to consult now? working note + prediction current word + previous working note
The LSTM cell. Long-term memory rides the belt across the top, modified only where the gates allow: the forget gate can erase, the input gate can add, and the output gate taps the belt to build this step's working note. Each gate looks at the current word and the previous note to decide how far to open.

For the curious: each gate is computed as sigmoid(W·[note, word] + b), and the sigmoid from ML Basics Part 9 is exactly why: it squashes any number into the 0-to-1 range, which is precisely what a valve needs. The belt update is belt_new = forget × belt_old + input × candidate, when the forget gate sits near 1, the belt carries memory forward essentially untouched, which is what defeats the vanishing problem.

A sentence walking through the gates

Follow the LSTM through "Tom went to the store. He bought apples." The interesting question: when the network reaches "He", how does it know who "He" is? "Tom" was six words ago, ancient history for a fading notepad, but not for a belt. Step through and watch the belt hold the subject while the working note flits from word to word. (The gate values here are hand-picked to tell the story; a real LSTM learns them.)

the belt (long-term memory): how strongly "Tom = the subject" is being held

Press "Read next word" to begin.

Compare the green belt with the notepad from Part 1's demo: the plain RNN's memory of the first word decayed at every single step no matter what, while the belt holds "Tom" at nearly full strength for the entire text, because holding was a decision, not an accident. If the story moved on to a new subject, a trained LSTM would close the forget gate on Tom and write the newcomer in his place.

What's next

LSTMs could finally read long text without losing the plot, and they powered translation, speech recognition, and autocomplete through the 2000s and 2010s. But reading is only half a conversation. To translate a sentence, or answer a question, a network has to read everything in, and then write something new out. The next part covers the elegant two-network design that does this, and the bottleneck at its middle that set the stage for the biggest idea in modern AI.