Where we are so far

Five parts in, the whole of training now fits in a short list. Everything below has been built up piece by piece, and it is worth seeing it in one place before we zoom in on the one number this part is about.

  1. A model is a set of numbers. For a straight line that is two of them, a weight and a bias (Part 2). Every prediction the model will ever make comes from those numbers.
  2. Nobody hands you the right values, so you begin with a guess, often a random one (Part 3).
  3. A loss function scores that guess. Run every data point through the line, measure how far each prediction lands from the truth, and boil it down to one number (Part 4).
  4. A gradient says which way to move. For each knob, one number answering "would nudging this up make the loss better or worse, and how sharply?" (Part 5).
  5. Take a small step downhill, and repeat. Adjust every knob a little against its gradient, score again, and keep going until the loss stops improving.

Step 5 hides a question the previous parts have quietly deferred: how big is "a small step"? The gradient tells you the direction and how steep the ground is, but not how far to walk. That decision is a separate number you have to supply yourself, and it is the subject of this part.

The knob that sets the step size

In the gradient descent update rule:

w ← w - learning_rate × gradient

the learning_rate (often written η or lr) is a small positive number that controls how big each step is. In the simplest form of gradient descent it stays constant for the whole training run, and you pick it before training starts.

It is worth being clear about what kind of number this is. The weight and the bias are learned: training discovers them for you. The learning rate is not learned, it is chosen, before training begins, by you. Numbers like that are called hyperparameters, settings that govern how the learning happens rather than being an outcome of it. The next part introduces two more (how many examples each step looks at, and how many passes to make over the data), but this is the one that matters most.

Choosing the right learning rate is one of the most important (and annoying) practical decisions in machine learning.

Too small: slow but safe

weight loss the bottom (the goal) start each loop is one tiny step
A small learning rate takes many tiny steps, drawn here as a red coil winding down the loss curve, one loop per step. It is safe and steady, but reaching the bottom can take a great many iterations.

If the learning rate is very small, each step moves only a tiny distance down the loss curve. Training will eventually converge, but it can take thousands of iterations to get there. In practice, for large models, this means hours or days of extra computation.

Too large: oscillation and divergence

If the learning rate is too large, each step overshoots the minimum. Instead of landing near the bottom of the curve, you land on the other side, and the next step overshoots back. The weight bounces from side to side around the minimum, and each bounce wastes most of the step. Push the rate a little higher and each overshoot lands further from the bottom than the last: the loss grows with every step instead of shrinking, the weight flies off towards infinity, and training has diverged. In real training this shows up as a loss that suddenly reads as NaN ("not a number"), which is the computer's way of saying the numbers got too big to hold.

Interactive demo: compare learning rates

The animation below runs gradient descent on the same dataset (y = 2x + 1) with four different learning rates at once. Watch how each one traces a different path on the MSE loss curve.

Four simultaneous runs: LR = 0.005 (slow) · LR = 0.05 (good) · LR = 0.125 (bouncing) · LR = 0.14 (diverging)

What to look for

In the slow panel (LR = 0.005), each step is tiny. The trajectory creeps along the bottom of the curve. It will eventually converge but takes many iterations.

In the good panel (LR = 0.05), steps are larger. The trajectory moves efficiently towards the minimum in far fewer iterations.

In the bouncing panel (LR = 0.125), steps overshoot the minimum. The trajectory bounces back and forth: weight = 0 overshoots to weight ≈ 3.75, then back to about 0.5, then out again. It does settle in the end, because each bounce is slightly shorter than the last, but it takes far longer than the good run to get there.

In the diverging panel (LR = 0.14), each bounce is slightly longer than the last. Within a few steps the weight has left the chart entirely and the loss readout is climbing into the millions. For this dataset the dividing line sits at about 0.133: below it every run eventually settles, above it every run blows up. That line is different for every dataset and every model, which is exactly why nobody can hand you the right learning rate in advance.

How a learning rate actually gets chosen

Since no formula gives the right value, people find it the same way you just did: by trying. The usual routine is to pick a handful of rates spread across a wide range, say 0.001, 0.01, 0.1 and 1, run each for a short while, and watch what the loss does over the first few hundred steps. A rate that makes the loss shoot up or turn into NaN is too big. A rate that barely moves it is too small. Somewhere between them is a rate that pulls the loss down quickly and keeps it falling, and that is the one to keep, perhaps after a second, finer round of trials around it. (Reading the loss over the course of a run, rather than against the weight, is a chart of its own, and the next part introduces it.)

One simple refinement is used almost everywhere: instead of one fixed rate, start large and shrink it as training goes on. Big steps early, when the guess is far from the bottom and the slope is steep, and small careful steps later, when it is close and a big step would only bounce. This is called a learning-rate schedule or decay, and it gives you the best of the slow and the good panels above without the worst of either.

Modern optimisers such as Adam, RMSProp and AdaGrad go one step further and adjust the rate automatically as they go, often using a different effective rate for every single weight, based on how the gradients for that weight have been behaving. You will see those names in the code of almost any real training run. Even so, every one of them still asks you for a starting learning rate, so the balancing act in this part never entirely goes away.

Next: Batches, Epochs and the Training Curve: why real training never looks at the whole dataset at once, what an epoch is, and how to read the chart that every training run produces.