Gradient Descent

We know that we need to find the right values for weight and bias. We know how to measure how wrong we are (the loss). Now the question is: how do we use the loss to find better values?

The answer is gradient descent. And here is something worth saying plainly: when people talk about "training a model", this is what they mean. Training is nothing more mysterious than the process of finding good values for the weights and biases.

The training loop

Gradient descent turns training into a simple loop. The steps below are numbered to match the diagram:

1. Set the initial weight and bias. We have already met this step: we simply start with some randomly chosen values. The line will fit badly at first, and that is fine.
2. Evaluate the model. Measure how wrong the current line is by computing its loss score, the MAE or MSE from the last part.
3. Is the model good enough? If the loss is small enough, stop. If not, carry on.
4. Tweak the weight and bias. Nudge them in a direction that lowers the loss, then go back to step 2. (Working out that direction is what the rest of this article is about.)
5. Success. Once the model is good enough, training is done.

Steps 2 to 4 are the part that repeats. Each pass through them is one iteration. (You will also meet the word epoch in ML libraries: an epoch is one full pass through the entire training dataset. In our tiny examples every iteration uses all the data, so the two happen to be the same thing here, but on big datasets a single epoch is made up of many iterations.) Think of it like training for a marathon: you don't wake up one morning and run 26 miles. You practice every day, get a little better, and gradually build towards your goal.

One label on the diagram deserves a note before we move on: the arrow that loops back is marked backpropagation. That is the name of the technique used to work out which way to tweak each weight and bias when a model has many layers of them. For our single-line model the tweak is simple enough to compute directly, as you are about to see, and we will come back to what "propagating backwards" means when we get to networks of neurons at the end of the series.

How do we know which direction to tweak?

This is the clever part. The loss curve is shaped like a U (a bowl): the loss is high out on the sides and lowest at the single best weight and bias, right at the bottom. Gradient descent wants to reach that bottom. But how does the algorithm know which direction is "downhill" from where it currently sits?

The answer is the gradient, the slope of the loss curve at the current point.

To turn "which way is downhill" into an actual number, we use a partial derivative of the loss with respect to the weight, written ∂MSE/∂w. The word partial means we wiggle only the weight and hold the bias still, then ask: if we nudge the weight up by a tiny amount, does the loss rise or fall, and how fast? That is exactly the slope of the loss curve in the weight direction, and its sign points us the way that would lower the loss.

When the loss function is MSE, that slope works out to this (same notation as the last part: yᵢ is the actual value, ŷᵢ the predicted one, n the number of data points):

The MSE in the symbol ∂MSE/∂w is a label, not an instruction: it names which loss we are taking the slope of. Read it as "the slope of the MSE, in the w direction". Pick a different loss and you would write ∂MAE/∂w instead, and get a different expression on the right.

This is not the MSE formula. The one from the last part, which squares each error and averages, answers "how wrong is this guess?" and returns a loss, like the 76.33 we computed there. The formula above answers a different question, "which way should the weight move, and how urgently?", and returns a slope. Same data, same guess, two questions, two formulas. (Change the loss function and this slope formula changes too, which is one more reason the choice of loss matters.)

Reading the formula piece by piece

It looks dense, but it is four small ideas stacked together. Working from the inside out:

Piece	What it does
(ŷi − yi)	The error for one data point. The same error you have been computing since the last part: how far this point’s prediction lands from its true value. Negative means the line is below the point.
xi	Multiply that error by the point’s own input. This is the part that makes it a slope for the weight specifically. Changing the weight tilts the line about the y-axis, so a point far out at xi = 100 swings a long way while a point at xi = 1 barely moves. Each point’s vote is scaled by how much leverage it has.
Σ from i=1 to n	Add up those contributions across every data point. Every row gets a say. Points pulling the line up and points pulling it down partly cancel, and what survives is the net direction.
(2/n)	Average it, with a 2 attached. Dividing by n keeps the answer the same size whether you have 6 rows or 6 million. The 2 is the fossil left behind by MSE’s squaring, explained just below; it only scales the result, and in practice it is absorbed into the learning rate anyway.

Wait, where did the square go?

A fair question, since MSE is built on squared errors and there is not a square in sight above. The answer: the square has not disappeared, it has been used up. Taking a slope consumes it, and leaves a trace behind.

The intuition without any calculus: squaring means a doubled error costs four times as much. So near any given point, the rate at which the loss grows is proportional to how big the error already is, twice as big, twice the rate. That "twice" is the 2 in the formula, and the error itself is the (ŷᵢ − yᵢ) term. The square has turned into a multiplication.

For the curious: this is the chain rule doing two small jobs. First, differentiating the square: the derivative of (error)² is 2 times the error, which is where both the 2 and the bare, un-squared (ŷᵢ − yᵢ) come from. Second, asking how the error itself responds to the weight: since the prediction is ŷᵢ = w·xᵢ + b, nudging w up by a tiny amount changes ŷᵢ by xᵢ, which is where the xᵢ in the formula comes from. Multiply the two together and average, and you have the formula above. The bias version is the same story, except nudging b changes every prediction by exactly 1 rather than by xᵢ, which is why the xᵢ is simply absent there.

So the squaring in MSE still shapes everything: it is the reason badly-fitted points pull harder on the result, and the reason the gradient shrinks smoothly to zero as the fit improves. It is just no longer written as a square.

The single number that comes out is the overall slope: which way, and how steeply, the loss changes as we change the weight.

That single number, the slope ∂MSE/∂w, also called the partial derivative or simply the gradient, is all we need to decide the move. Read its sign:

• If the gradient is positive, nudging the weight up would raise the loss, so we go the other way and decrease the weight.
• If the gradient is negative, nudging the weight up would lower the loss, so we increase the weight.
• If the gradient is near zero, the loss barely changes in either direction, we are at the bottom, and there is nowhere better to go.

So we always step in the direction opposite to the gradient. The actual amount we add to or subtract from each value is that gradient scaled down by a small learning rate:

w_new = w - learning_rate × ∂MSE/∂w
b_new = b - learning_rate × ∂MSE/∂b

A worked step

Now run those four pieces on real numbers, the way we worked through the loss itself in the last part. We will use the y = 2x + 1 data and a poor guess of weight = 1 (with the bias fixed at 1), so the line is ŷ = 1·x + 1, too shallow.

Piece 1, the error for each point. Run every feature through the line and compare with the truth:

xᵢ	ŷᵢ (1·xᵢ + 1)	yᵢ	error (ŷᵢ − yᵢ)
1	2	3	-1
2	3	5	-2
3	4	7	-3
4	5	9	-4

Every error is negative, which is already a hint: the line sits below every point, so it presumably needs to rise. Hold that thought loosely, though. Reading a table by eye is exactly the habit we are trying to replace, and it only works because this example is tiny and unusually tidy. With a real dataset the errors would be a mix of positive and negative, running into millions of rows, and no glance would tell you anything. So let us ignore the hint and turn the handle: the point of the next three steps is that the arithmetic reaches the same conclusion on its own, and, unlike our eyes, also tells us how big a step to take.

Piece 2, multiply each error by its xᵢ. This is the leverage step. The point at xᵢ = 4 is furthest out, so tilting the line moves it four times as much as the point at xᵢ = 1, and its vote counts four times as heavily:

xᵢ	error	error xᵢ
1	-1	-1
2	-2	-4
3	-3	-9
4	-4	-16

Piece 3, add them up. One number for the whole dataset:

Σ (error xᵢ) = -1 + (-4) + (-9) + (-16) = -30

Piece 4, average it (with the 2). Here n = 4, so the factor is 2/4 = 0.5:

∂MSE/∂w = 0.5 × (-30) = -15

The gradient is -15. Two things to read off it. Its sign is negative, meaning "the loss falls as the weight rises", so we should move the weight up, which is exactly what our eyeball hint suspected, now arrived at mechanically and available to a computer that cannot squint at a table. Its size, 15, says the loss is changing steeply here, we are far from the bottom and can afford a decent-sized step. Later, near the bottom, this same calculation will return something close to zero, which is the signal to stop.

Now convert the gradient into an actual move, scaled by the learning rate, say 0.05:

Δw    = - learning_rate × ∂MSE/∂w = - 0.05 × (-15) = +0.75
w_new = 1 + 0.75 = 1.75

One step has carried the weight from 1 to 1.75, well on its way to the true value of 2, and the loss drops. Feed the new weight back into the same formula, repeat, and gradient descent walks the weight down to the bottom of the U. (The bias is updated the same way each step, using its own partial derivative.)

The learning rate

Look again at the small number the gradient gets multiplied by in the update: the learning rate. It is what turns a raw gradient into an actual step.

The gradient on its own can be large. In the worked step above it was -15, and moving the weight by a full 15 would fling it far past the bottom of the curve. So we do not step by the whole gradient. We keep its direction (its sign) but shrink its size, multiplying it by a small learning rate such as 0.01 or 0.05:

step  = learning_rate × gradient
w_new = w - step

Multiplying by the learning rate scales the gradient down to a safe, controlled amount, and subtracting that step nudges the weight the right way, downhill towards lower loss. In the worked step, 0.05 × -15 = -0.75, and 1 - (-0.75) moved the weight up from 1 to 1.75.

The value of the learning rate is a balancing act: too large and the steps overshoot the bottom and bounce around, too small and training crawls. Choosing it well matters enough to be the subject of the whole next part.

What about the bias?

Until now we quietly held the bias fixed and only moved the weight, which is what let us draw the loss as a simple 2D curve. In a real model the bias is a free parameter too, and it gets its own partial derivative:

It is almost the same as the weight’s gradient, only with no xᵢ in it. That makes sense: nudging the bias shifts every prediction up or down by the same amount, whatever xᵢ is, so each point’s error counts equally.

From a loss curve to a loss surface

Letting the bias move as well does something worth pausing on, because it changes the picture we have been carrying since the last part.

Up to now the loss has been a curve: one knob along the bottom, the loss going up the side, a valley to slide into. That picture was only available because we froze the bias. With the bias frozen, each weight had exactly one loss value, so the results fitted on a flat page.

Now there are two knobs to set, and every combination of them has its own loss. Weight 2 with bias 0 gives one value; weight 2 with bias 1 gives another. So the loss can no longer be drawn against a single line of weights. It needs a whole floor of possibilities, with weight running one way and bias the other, and above every point on that floor sits the loss for that particular pair. What you get is not a curve but a loss surface, a landscape: two axes for the two knobs, and height for how wrong that combination is.

And for a straight-line model with MSE, that landscape has a very friendly shape, a bowl. Push the weight too far in either direction and the loss climbs. Push the bias too far either way and it climbs too. Every direction leads uphill except towards one spot: the single lowest point of the bowl, which is the best weight and bias together.

Two things carry over unchanged from the curve, which is the reassuring part:

• The goal is still the bottom. Before we were looking for the lowest point of a valley; now we are looking for the lowest point of a bowl. Same job.
• We still only ever read the local slope. The gradient now has two components, ∂MSE/∂w and ∂MSE/∂b, and together they say which way is downhill on the surface, exactly as the single slope did on the curve.

The one genuinely new thing is that a step is now a move in two directions at once: a little along the weight axis, a little along the bias axis, like walking diagonally down a hillside instead of sliding along a groove.

Something else follows from this that matters later. If two knobs make a surface, three would need a space, and four cannot be pictured at all. A model with a million weights has a "surface" in a million dimensions, which nobody can draw or imagine. That sounds alarming and turns out not to matter one bit, because gradient descent never looks at the whole landscape anyway. It only ever asks for the slope where it currently stands, and that question has an answer no matter how many dimensions there are. Part 8 returns to this once the landscape stops being a tidy bowl.

Gradient descent now works in two dimensions at once. At each step it computes both partial derivatives, ∂MSE/∂w and ∂MSE/∂b, and nudges the weight and the bias together, rolling the guess a little further down the bowl towards that lowest point.

When to stop

Training stops when one of three conditions is met:

Convergence: the happy ending. The loss is no longer improving: either the loss itself is near zero, or the parameters are barely changing between iterations, or the gradient has flattened out near zero. We have found the bottom of the curve.

Early stopping: something is wrong. The loss keeps falling on the data the model is trained on, but when we check it against a small stash of data we deliberately held back, its predictions there start getting worse. That means the model has begun memorising its training data instead of learning the underlying pattern, so we stop early and rethink. (This idea, called overfitting, and the held-back data used to catch it, get a whole part of their own later in the series.)

Maximum iterations: a safety net. Set a hard limit to prevent infinite loops and also useful for benchmarking (comparing two models at the same number of training steps).

Weight and bias together, step by step

Let us watch the full loop run. Starting from a blank slate of weight = 0 and bias = 0 on the same y = 2x + 1 data, at every step we compute both partial derivatives and nudge each value by the learning rate (0.05). Here are the first few iterations:

Step	weight	bias	loss (MSE)	∂MSE/∂w	∂MSE/∂b
0	0.00	0.00	41.00	-35.0	-12.0
1	1.75	0.60	1.13	-5.75	-2.05
2	2.04	0.70	0.04	-0.92	-0.41
3	2.08	0.72	0.01	-0.13	-0.14

Read the first row like this: both gradients are large and negative, so each update pushes its value up. The weight becomes 0 - 0.05 × (-35) = 1.75 and the bias becomes 0 - 0.05 × (-12) = 0.60. Those are the numbers on the next row, where we work out the gradients again and repeat.

The loss collapses from 41 to under 0.05 in just two steps. The weight snaps close to 2 almost at once, while the bias climbs towards 1 more gradually. Left to keep running, gradient descent settles both at the values that make the loss as small as it can be. That is the whole algorithm: find the slope for each parameter, take one small step downhill, and repeat.

Interactive demo: gradient descent in action

The animation below shows gradient descent running on a simple dataset (y = 2x + 1, 4 data points). The left panel shows the predicted line chasing the true line. The right panel shows the gradient descent trajectory on the MSE loss curve.

Press Start to begin. Watch how each step moves the weight slightly closer to the true value of 2.

Then experiment with the learning rate box above the buttons. Set it to 0.01 and run: the weight creeps towards 2 in tiny, steady steps, taking many iterations to get there. Now set it to 0.13 and run again: the steps are now so big that the weight shoots past the minimum and bounces back and forth across it, taking far longer to settle (a little higher still and it would fly off and diverge). Getting this number right is the whole subject of the next part.

What to notice

• The red dot on the loss curve moves step by step towards the minimum at weight = 2.
• The tangent line (dashed red) shows the gradient at the current point, steep far from the minimum, nearly flat when close.
• The predicted line (red) in the left panel converges towards the true line (black dashed).
• Try changing the learning rate. A smaller value takes more steps but is stable. A larger value can overshoot. See the next part for details.

---

Next: The Learning Rate: why the size of each step matters so much, and how too large or too small a learning rate can wreck or stall training.