Reading in Order: Networks with Memory

The feedforward network from the last part lives by one iron rule: signals flow one way, numbers in, prediction out, and then the network forgets you were ever there. Show it the same input twice and you get the same answer twice. For predicting a house price from its floor area, that is exactly what you want.

Language breaks this completely. Read these two sentences:

The film was good. Nobody could honestly say the film was good.

The last word is identical, but the meaning has flipped, and the only difference is what came earlier. Words do not carry meaning alone; they carry meaning in order. The same is true of anything that unfolds over time: share prices, heartbeats, weather. To predict tomorrow's house price you cannot just look at today's, you need the trend of the past weeks. A model that treats every input as a fresh start has no way to see a trend at all.

So the question this whole series answers is: how do you build a network that can read? Not glance, read, one word after another, carrying along an understanding of everything so far.

The idea: reading with a notepad

Think about what you do when you read. Your eyes take in one word at a time, but you are not starting from scratch at each word. You carry a running summary in your head, who the characters are, what the sentence is about, whether that "not" three words ago is still in force, and each new word updates that summary.

A recurrent neural network (RNN) copies this trick with one small addition to the neurons you already know. A neuron from ML Basics takes an input, multiplies by a weight, adds a bias, and squashes the result through an activation function. A recurrent neuron does the same, but with a second input: its own output from the previous step.

That looped-back value is called the hidden state, and it is the network's notepad. At every word, the network does two things:

1. Update the notepad: blend the new word with the current note to produce an updated note.
2. Make a prediction from the updated note, if one is needed at this point.

Then it moves to the next word and does the same two things again. The notepad is what connects word one to word twenty: by the time the network reaches "good" in "nobody could honestly say the film was good", the scepticism from "nobody" is still written on the pad.

The stretched-out picture on the right is called unrolling, and it is the honest view of what happens at run time: the loop is replayed once per word, however many words there are.

One rule for every word

Here is the detail that makes RNNs elegant rather than enormous. Look at the unrolled picture: three steps, three boxes. You might expect each box to have its own weights and biases. It does not. Every step uses the exact same weights, one shared rule for "how to blend a new word into the note", applied over and over.

This buys two things. First, thrift: a sentence of 100 words does not need 100 sets of knobs, it needs one. The network stays the same size no matter how long the text gets. Second, and more subtle: because the same rule must work at every position, the network is forced to learn something general about how words update meaning, "if this word arrives after that kind of note, adjust it like so", rather than memorising what tends to happen at position 17. That is what lets one trained RNN handle sentences of any length, including lengths it never saw in training.

For the curious: the notepad update is one line of maths, and every piece of it is from ML Basics. If x is the current word (as numbers), h_old the previous note, then h_new = tanh(Wx·x + Wh·h_old + b). Two weights and a bias, passed through the tanh activation from Part 9, the only novelty is that the neuron's own previous output, h_old, is one of its inputs. The prediction, when needed, is another plain neuron reading the note: y = Wy·h_new + c.

Watch a network read

The toy below is a one-number RNN: its entire notepad is a single value between -1 (this review feels negative) and +1 (positive). Each word nudges the note; the note also carries forward, faded slightly, from the previous word. Step through the three reviews and watch the meter.

Two things to notice. In Review B, watch the note climb through "a great start", then get dragged down, the final reading remembers both halves of the review, which no word-by-word tally could do. Then compare the ends of Review A and Review C: Review C finishes with "stunning", the most positive word in the whole vocabulary, yet its final note is well below Review A's, because the dull opening is still on the notepad. Same rule at every word, but history changes what each word does. That is memory.

Where "recurrent" comes from

The name has a lovely backstory: it comes from anatomy, not computing. In the late 1800s, Camillo Golgi developed a way to stain nervous tissue and saw it as one continuous web. Santiago Ramón y Cajal, using Golgi's own stain, showed the opposite, the brain is made of separate, individual cells, neurons, passing signals in one direction. (They shared a Nobel Prize while disagreeing with each other, which remains excellent scientific sportsmanship.) Cajal noted "recurrent semicircles" in the cerebellum as early as 1901. It was his student, Rafael Lorente de Nó, who in 1933 formally described recurrent loops in the cortex: circuits where a neuron's output travels around and comes back to influence its own layer again. The artificial networks in this article borrow both the idea and the word: output that circles back as input.

What's next

RNNs can read. But hand one a long paragraph and something troubling happens: by the time it reaches the end, the beginning has faded off the notepad almost entirely, and training makes it worse. The next part is about why RNNs forget, why the fix needed gates, and the ingenious cell, the LSTM, that carried language AI for two decades.