The linear model y = wx + b is powerful, but it is fundamentally limited: it can only represent straight lines. Real-world data might curve, twist, or look like someone had too much coffee before plotting it.
Straight-line thinking, "each extra unit of input adds the same amount to the output", fails constantly once you leave textbook examples:
- Population growth. A colony of bacteria does not add a fixed number of cells per hour; it doubles. Growth starts imperceptibly, explodes, and then flattens off as food runs out, giving the S-shaped curve you saw in every epidemic chart of the 2020s.
- A thermostat or a heater. Turn the dial up and the room warms, up to a point. Beyond it the heater is already at full power and the temperature stops responding, no matter how far you keep turning. Real sensors and actuators nearly all behave this way: linear in the middle, flat at the extremes, because physical things have limits.
- Braking distance. Double your speed and you do not need double the distance to stop, you need roughly four times as much. The relationship bends upward, which is why speed limits matter more than intuition suggests.
- Medicine dosage. Too little has no effect, the right amount works, and too much is toxic. The best outcome sits in the middle, so the curve rises and then falls, something no straight line can express at all.
- Daily temperature. Warm at midday, cold at midnight, warm again tomorrow. Anything that cycles is hopeless for a line, which can only ever head in one direction forever.
- Sensor readings in general. A battery's voltage as it discharges, a microphone's response to loudness, an engine's efficiency across its rev range, all curve, and most flatten out or reverse somewhere.
Some of these bend gently, some saturate at a ceiling, some rise then fall, some repeat. A straight line, with its single fixed slope, can approximate any of them over a small enough stretch and none of them across the full range.
A quick refresher, and where it breaks
Everything so far has been built on one small table. Hours studied on the left, exam score on the right:
| Hours studied (feature) | Exam score (target) |
|---|---|
| 1 | 53 |
| 2 | 60 |
| 3 | 64 |
| 4 | 71 |
| 5 | 75 |
| 6 | 81 |
Every hour of study buys roughly five and a half more points, all the way along. That steady, repeating step is exactly what y = w·x + b encodes, which is why the line fitted this data so well: the best one, y ≈ 5.49x + 48.1, misses each student by about a point.
Now here is a table of exactly the same shape, from a bacteria colony counted once an hour:
| Hours elapsed (feature) | Cells in the colony (target) |
|---|---|
| 0 | 10 |
| 1 | 20 |
| 2 | 40 |
| 3 | 80 |
| 4 | 160 |
| 5 | 320 |
| 6 | 640 |
Two columns of numbers, same as before. But read down the right-hand column: the first hour adds 10 cells, the last adds 320. There is no "amount added per hour" to put in w, because the amount added keeps changing. The straight-line model has no way to say doubles every hour.
Watch what happens when you fit one anyway:
And note what has not gone wrong. Gradient descent did its job perfectly, that really is the best line available; there is no better weight and bias to find. The failure is not in the training, it is in the model: we asked for a straight line, and the answer is not a straight line. No amount of searching fixes a shape that cannot express the pattern.
So the fix has to change the shape of the model itself. There are three ways to do that.
Three ways to go non-linear
When a straight line won't fit, there are a few options.
Use a non-linear function as the model. Instead of y = w·x + b as your model, pick a formula that already has the right shape and fit that. If the data looks like a wave, make the model y = w·sin(x) + b; if it is S-shaped, y = w·tanh(x) + b; if it grows by doubling, y = w·e^x + b. You are still hunting for a weight and a bias, still scoring guesses with a loss, still rolling downhill, everything from Parts 2 to 6 applies unchanged. The only thing that has changed is the shape the two knobs are stretching and shifting: a wave or an S instead of a straight line.
Add higher-order terms. Instead of just x, include x², x³, and so on, each with its own weight (all sharing a single bias). This is the trick from Part 3: still regression, but it can fit far more interesting shapes than a straight line.
Both of these work, but each one commits you to a specific shape picked in advance. The third option is the trick that powers neural networks:
Activation functions keep the simple, trainable linear structure and layer the non-linearity on top. The idea:
- Compute the linear output:
z = wx + b - Pass it through an activation function:
output = activation(z)
The activation function has no weights or biases of its own. It is just a fixed mathematical transformation. But it makes the output non-linear, and once you chain non-linear units together, you can model surprisingly complex patterns.
Nothing else about training changes
This is the point worth holding on to, because it is easy to assume that a non-linear model needs a new kind of training. It does not. Look back at the five-step loop from Part 6 and mark what an activation function touches:
| Step | Linear model | With an activation function |
|---|---|---|
| 1. Make a prediction | ŷ = w·x + b |
ŷ = activation(w·x + b) ← the only change |
| 2. Measure the loss | compare ŷ with y, average it |
identical |
| 3. Find the gradients | which way should w and b move? |
identical in purpose |
| 4. Take a step | w ← w − learning_rate × gradient |
identical |
| 5. Repeat | until the loss stops improving | identical |
One line of the five. The loss function does not know or care how ŷ was produced, it only sees a predicted number and a true one. Gradient descent still nudges the same two knobs in the same direction for the same reason. The weight still controls steepness, the bias still shifts things along. You have not learned a new training algorithm; you have swapped the formula that turns x into ŷ, and left everything downstream of it alone.
(The gradient calculation does have to account for the extra step, since the prediction now passes through one more function on its way out. That is a change in the arithmetic, not in the idea, and it is the same chain rule already met in Part 5.)
This little unit, a weight, a bias, and an activation function, has a famous name: it is a neuron, the basic building block of every neural network. The name is borrowed from biology, and the analogy is loose but helpful: like a brain cell, it receives a signal (x), decides how strongly to respond (w and b), and either fires or stays quiet (the activation function). Keep that picture in mind for the rest of this article; the next part is about what happens when neurons team up.
Neurons are so central to machine learning that they have a standard way of being drawn, and it is worth learning to read it now, because every neural network diagram you will ever see is built from copies of this picture:
x travels along a connection whose label w is the weight it gets multiplied by; the bias b enters from below; the circle is the neuron itself, which adds everything up (wx + b) and pushes the result through its activation function (the little hockey-stick, ReLU here); the arrow leaving the circle carries the output. When you see a web of circles and arrows later in the series, it is just many of these wired together.ReLU (Rectified Linear Unit)
The simplest and most widely used activation function:
ReLU(z) = max(0, z)
It strips away the negative half of the line, leaving zero for any negative input and the value itself for positive inputs. The result looks like a hockey stick.
To see what this does inside a neuron, run some numbers through one. Take a neuron with w = 2 and b = -4, so its raw output is z = 2x - 4, and follow each input through both steps:
| x (input) | z = 2x - 4 | ReLU(z) (output) |
|---|---|---|
| -1 | -6 | 0 |
| 0 | -4 | 0 |
| 1 | -2 | 0 |
| 2 | 0 | 0 |
| 3 | 2 | 2 |
| 4 | 4 | 4 |
| 5 | 6 | 6 |
The left two columns are plain Part-2 material: a straight line. The third column is where the activation earns its keep. For every input up to 2 the neuron says exactly 0, total silence, and from 3 onwards it passes the line through untouched. The weight and bias chose where the neuron wakes up (at x = 2, where z crosses zero) and how steeply it responds after that; ReLU is what lets it sleep through everything else.
Where it's useful: Sensor values that can't go negative, image pixel intensities, anything where "nothing happened" is meaningfully different from "something happened."
Leaky ReLU
LeakyReLU(z) = z if z > 0, else α × z (typically α ≈ 0.01)
Like ReLU but allows a small negative slope for negative inputs. Think of a diode: it blocks negative voltage, but in reality a tiny leakage current still flows.
It is useful when negative values still carry weak but useful information (a financial loss still tells you something about risk). It also fixes a real training hazard called the dying ReLU problem: if a plain ReLU neuron's input becomes negative for every data point, its output is always zero, so nudging its weight or bias no longer changes anything, the gradient is zero, and gradient descent can never wake it up again. The neuron is dead for the rest of training. The leak keeps a faint signal flowing, so the neuron can always recover.
Clipped ReLU
ClippedReLU(z) = max(0, min(cap, z))
Clips the output at a maximum value. Useful when the output has a physical ceiling: a sensor's maximum reading, a network link's maximum throughput, screen brightness between zero and full.
Sigmoid
sigmoid(z) = 1 / (1 + e^{-z})
(The e here is Euler's number, ≈ 2.718, a constant that appears throughout maths; all you need to know is that e^{-z} shrinks rapidly towards zero as z grows.)
Sigmoid squashes any input to a value between 0 and 1, with an S-shaped curve. It models phenomena that start slow, accelerate, then saturate: population growth, epidemic spread, probability outputs.
Tanh (Hyperbolic Tangent)
tanh(z) = (e^z - e^{-z}) / (e^z + e^{-z})
Like sigmoid but ranges from -1 to +1. Useful when you need a symmetric output centred at zero.
Interactive demo
Use the controls below to explore each activation function. The thin dark line is the raw linear output z = wx + b, exactly the straight line you know from Part 2. The thick dark-red line is what comes out after the activation function is applied. Adjust weight and bias to see how the activation shapes the output.
activation: ReLU · z = 1.0x + 0.0
What to try
- Pick ReLU, set the weight to 1 and slide the bias around. The kink (where the neuron "switches on") slides left and right: the bias decides where the neuron starts responding.
- Now slide the weight. The active side gets steeper or shallower: the weight decides how strongly it responds. With a negative weight, the neuron responds to small inputs and ignores large ones, the hockey stick flips.
- Switch to sigmoid and push the weight up to 4: the gentle S sharpens towards a step. Weight controls how abrupt the transition is; bias slides the step along the x-axis.
- Compare ReLU and leaky ReLU with a negative weight and bias: watch the leaky version keep a faint slope where plain ReLU flatlines at zero. That faint slope is exactly what keeps a dying neuron trainable.
One neuron with an activation function can bend a line once, place the bend anywhere, and control the slope on each side. That may not sound like much, but it is the only ingredient we were missing.
Next: Neurons, Chaining, and Specialisation: what happens when neurons connect, in series and in parallel, and how a handful of them can learn to draw a sine wave.