Every architecture in this series, the networks that read, remember, translate, and chat, is built from one and the same basic component. Before meeting the fancy ones, it pays to know the original: the feedforward neural network, the simplest, oldest, and still most common way to organise neurons into something useful. (If you finished ML Basics, you have already built one: the "5 parallel neurons feeding an output neuron" from Part 10 was exactly this, just never named.) This part names the parts, walks the machinery by hand, and tells the story of the tiny logic puzzle that made hidden layers famous.
The shape: neurons in layers
A feedforward network organises neurons into layers:
- The input layer is not really neurons at all, just your features lined up (square footage, hours studied, pixel values), one slot each.
- One or more hidden layers of genuine neurons, each doing exactly what ML Basics Part 9 taught: weighted sum, plus bias, through an activation function.
- An output layer that produces the final answer, one neuron for a single predicted number, several for multi-way answers.
Two rules define the architecture. First, every neuron receives input from every neuron in the layer before it, which is why these layers are called fully connected (or dense). Second, and this is the rule in the name, signals flow strictly one way: input → hidden → output. No loops, no going back, no neuron ever hears its own output. The data is fed forward, hence feedforward. (You will also hear the historical name multilayer perceptron, or MLP.)
Counting the knobs is straightforward and worth doing once. Each connection carries a weight, and each neuron has a bias. For the 2–3–1 network above: the hidden layer has 2 × 3 = 6 weights plus 3 biases, and the output layer has 3 × 1 = 3 weights plus 1 bias. Total: 13 knobs, all found by the gradient descent from ML Basics Part 5. Scale the same arithmetic up and you get the parameter counts that make headlines: the layers are bigger, the recipe is identical.
The forward pass
Running the network, taking features in, producing a prediction out, is called the forward pass, and it is nothing more than the neuron arithmetic applied layer by layer: compute every hidden neuron from the inputs, then compute the output from the hidden values. Done. (Training, for the record, is the same story as ever: forward pass, measure the loss, send blame backwards with backpropagation, nudge all 13 knobs downhill, repeat.)
Let us run one by hand, and not with any old network, with a famous one.
The little puzzle that nearly killed the field
Consider the humblest possible task, called XOR ("exclusive or"). Two inputs, each 0 or 1. Output 1 if exactly one input is on, 0 otherwise: a light controlled by two switches at opposite ends of a staircase.
| x1 | x2 | wanted output |
|---|---|---|
| 0 | 0 | 0 |
| 0 | 1 | 1 |
| 1 | 0 | 1 |
| 1 | 1 | 0 |
In 1969, Marvin Minsky and Seymour Papert proved something embarrassing: a single neuron, no hidden layer, cannot compute XOR at all. Not "struggles with", cannot. You can see why with ML Basics Part 2 eyes: a lone neuron draws a straight line through the input space, everything on one side scores high, the other side low. Now plot the four corners: the two "output 1" cases, (0,1) and (1,0), sit diagonally opposite each other, with the "output 0" cases, (0,0) and (1,1), on the other diagonal. No single straight line can put the two 1s on one side and the two 0s on the other. Try it mentally; then feel free to give up, because it is provably impossible.
The result landed hard. Funding dried up, researchers left, and the episode became part of the first "AI winter". Which is ironic, because the cure was already conceptually available: a hidden layer. Hidden neurons build intermediate features, and with the right intermediate features, the impossible becomes trivial. Here is a complete XOR network, two hidden ReLU neurons, one plain output neuron, with weights chosen for clarity:
h1 = ReLU( x1 + x2 ) "how many switches are on?"
h2 = ReLU( x1 + x2 - 1 ) "are BOTH switches on?"
y = h1 - 2·h2 "count the switches, then heavily punish 'both'"
Walk all four rows through it:
| x1 | x2 | h1 = ReLU(x1+x2) | h2 = ReLU(x1+x2-1) | y = h1 - 2·h2 |
|---|---|---|---|---|
| 0 | 0 | 0 | 0 | 0 |
| 0 | 1 | 1 | 0 | 1 |
| 1 | 0 | 1 | 0 | 1 |
| 1 | 1 | 2 | 1 | 0 |
Perfect score, with five weights and a bias. Notice how it wins: neither hidden neuron is clever alone, h1 just counts, h2 just detects "both on", but the output layer combines them into something no single line could express. This is ML Basics Part 10's specialisation in its smallest possible form, and it is the entire argument for hidden layers, settled in one table.
Run it yourself
The network below is the XOR network, live. Drag the two inputs (or use the corner buttons) and watch every value flow left to right through the forward pass. The inputs are continuous, so you can also do something the truth table cannot: slide between the corners and watch the network's in-between opinion.
Try the four corner buttons first and check them against the table. Then slide both inputs to 0.5 each: the output hits its maximum, because h1 says "one switch's worth of on-ness" while h2 stays silent. Slide both toward 1 and watch h2 wake up and drag the output back down. The network has quietly built a tent over the input space, high along the diagonal where x1 + x2 ≈ 1, low at both "agreeing" corners, which is exactly the bump-building trick from ML Basics Part 10, in its two-neuron infancy.
For the curious: there is a remarkable theorem about these humble networks. The universal approximation theorem says a feedforward network with just one hidden layer can approximate essentially any well-behaved function as closely as you like, given enough hidden neurons. So why stack layers deep instead of wide? Because "enough" can mean absurdly many: the theorem promises existence, not efficiency, and says nothing about whether gradient descent will actually find those weights. Depth, reusing simple features to build complex ones, layer upon layer, gets there with exponentially fewer neurons in practice. Width is a warehouse; depth is an assembly line.
Where feedforward networks live today
On their own, feedforward networks remain the everyday workhorse for tabular data, predicting churn from customer attributes, prices from features, risk from measurements, anywhere the inputs are a fixed set of numbers with no particular ordering.
But their bigger role is as the standard building block inside larger architectures, which is why this series starts here. The transformer, the architecture behind ChatGPT and the destination of this series, contains a feedforward network inside every one of its layers, and those unglamorous blocks hold the majority of a modern LLM's weights. Learn this one architecture and you have learned most of the atoms of the giant ones.
And their defining rule, signals flow one way, no loops, is also their defining limit: a feedforward network takes a fixed-size input, has no memory, and does not know what it saw a moment ago. Feed it a sentence and it sees a bag of numbers, not a sequence. For language, that is disqualifying, and fixing it is where this story really begins.
Next: Reading in Order: Networks with Memory: what happens when you break the one rule, let a neuron's output loop back in, and give a network the ability to actually read.