Where we are so far

Five parts in, the whole of training now fits in a short list. Everything below has been built up piece by piece, and it is worth seeing it in one place before we zoom in on the one number this part is about.

  1. A model is a set of numbers. For a straight line that is two of them, a weight and a bias (Part 2). Every prediction the model will ever make comes from those numbers.
  2. Nobody hands you the right values, so you begin with a guess, often a random one (Part 3).
  3. A loss function scores that guess. Run every data point through the line, measure how far each prediction lands from the truth, and boil it down to one number (Part 4).
  4. A gradient says which way to move. For each knob, one number answering "would nudging this up make the loss better or worse, and how sharply?" (Part 5).
  5. Take a small step downhill, and repeat. Adjust every knob a little against its gradient, score again, and keep going until the loss stops improving.

Step 5 hides a question the previous parts have quietly deferred: how big is "a small step"? The gradient tells you the direction and how steep the ground is, but not how far to walk. That decision is a separate number you have to supply yourself, and it is the subject of this part.

The knob that sets the step size

In the gradient descent update rule:

w ← w - learning_rate × gradient

the learning_rate (often written η or lr) is a small positive number that controls how big each step is. In the simplest form of gradient descent it stays constant for the whole training run, and you pick it before training starts.

It is worth being clear about what kind of number this is. The weight and the bias are learned: training discovers them for you. The learning rate is not learned, it is chosen, before training begins, by you. Numbers like that are called hyperparameters, settings that govern how the learning happens rather than being an outcome of it. You will meet a few more later in the series, but this is the one that matters most.

Choosing the right learning rate is one of the most important (and annoying) practical decisions in machine learning.

Too small: slow but safe

weight loss the bottom (the goal) start each loop is one tiny step
A small learning rate takes many tiny steps, drawn here as a red coil winding down the loss curve, one loop per step. It is safe and steady, but reaching the bottom can take a great many iterations.

If the learning rate is very small, each step moves only a tiny distance down the loss curve. Training will eventually converge, but it can take thousands of iterations to get there. In practice, for large models, this means hours or days of extra computation.

Too large: oscillation and divergence

If the learning rate is too large, each step overshoots the minimum. Instead of landing near the bottom of the curve, you land on the other side, and the next step overshoots back. The weight oscillates back and forth around the minimum, never actually reaching it. With an extreme learning rate, the loss can actually increase with each step (divergence).

Interactive demo: compare learning rates

The animation below runs gradient descent on the same dataset (y = 2x + 1) with three different learning rates. Watch how each one traces a different path on the MSE loss curve.

Three simultaneous runs: LR = 0.005 (slow) · LR = 0.05 (good) · LR = 0.125 (oscillating)

What to look for

In the slow panel (LR = 0.005), each step is tiny. The trajectory creeps along the bottom of the curve. It will eventually converge but takes many iterations.

In the good panel (LR = 0.05), steps are larger. The trajectory moves efficiently towards the minimum in far fewer iterations.

In the oscillating panel (LR = 0.125), steps overshoot the minimum. The trajectory bounces back and forth: weight = 0 overshoots to weight ≈ 4, then back, then back again. The weight never settles.

Adaptive learning rates

In practice, manually picking a fixed learning rate is hard. Modern optimisers like Adam, RMSProp, and AdaGrad adapt the learning rate automatically: they start with a larger step and shrink it as they approach the minimum, or they use different rates for different parameters. But even with these methods, you still need to provide an initial learning rate to start from.

Next: Overfitting and Generalisation: what "good enough" really means, why a model can ace its training data and still fail in the real world, and how held-back data keeps it honest.