Every gradient descent step so far has used the whole dataset. To work out the slope in Part 5 we ran all four rows through the line, took the error on each one, and added the results up. Four rows, four errors, one slope. Easy.
Now picture the dataset a real model trains on: a million rows of house sales, or a billion sentences. One gradient descent step means running every one of those rows through the model, and then nudging the weights by a tiny amount. Then doing it all again for the next tiny nudge. Two things break at once. The dataset may not fit in memory, so you cannot hold it all at the same time. And even if it did, spending a full pass over a million rows to move each knob by a hair is a terrible rate of exchange. A model that needs ten thousand nudges would need ten billion row-evaluations.
Look at a slice instead
The fix is the one you would come up with yourself: do not add up all the errors, add up some of them. Take a random handful of rows, say 32 or 128, compute the slope from just those, take the step, then grab the next handful. Each handful is called a batch (or mini-batch), and the number of rows in it is the batch size.
This rewrites the vocabulary from Part 5 slightly, and the two words are worth keeping straight because every training log you ever read is written in them:
- An iteration (or step) is one batch through the model and one update of the weights. Same as before, just with fewer rows behind it.
- An epoch is one complete pass through the whole dataset, however many iterations that takes.
With a million rows and a batch size of 100, one epoch is 10,000 iterations, so training for three epochs means 30,000 nudges to the weights, each one paid for with only 100 rows of work. In our four-row examples the batch was the whole dataset, which is why iteration and epoch happened to mean the same thing there.
The two extremes have names of their own. Using the entire dataset for every step is full-batch gradient descent, the version this series has used until now. Using a single row per step is the original stochastic gradient descent, where stochastic just means "involving randomness", the randomness being which row you happen to pick. In practice almost everyone uses something in between, and the whole family gets called SGD regardless of the batch size. When you see SGD in a line of code, this is what it means: gradient descent fed one batch at a time.
At the end of each epoch the rows are shuffled so that the next epoch's batches are different. Otherwise the same rows would always travel together, and any quirk of one batch would be repeated every epoch instead of averaging out.
The price: a noisier slope
A batch's slope is an estimate of the slope you would get from the whole dataset. It is usually pointing roughly the right way, but not exactly, and a smaller batch gives a rougher estimate. So instead of gliding smoothly downhill, mini-batch training jitters its way down: mostly the right direction, with a random wobble on every step. Watch it happen below.
The dataset has 40 points scattered around the line y = 2x + 1. Choose how many rows each step looks at, press Run (each press is a fresh run from weight 0, bias 0 with the current choice), and watch two things: the path on the loss landscape (weight across, bias up, colour is the loss, exactly as in Part 5), and the loss over time in the strip underneath. The counters show iterations, epochs, and how many rows of work have been done.
Run all three settings and compare the strip. With all 40 rows the loss falls in a clean curve and lands on about 0.51, which is as low as it can go for this data (the points were scattered on purpose, so no line fits them exactly). With 4 rows the path jitters and the loss wobbles between about 0.5 and 0.75 for the rest of the run, never quite settling. With 1 row the wobble is wilder, and at times the loss on the full dataset briefly rises even though every single step was downhill for its own row.
Now look at the rows-processed counter. After 300 steps, full-batch training has done 12,000 rows of work, the mini-batch run 1,200, and the one-row run 300, and all three found roughly the same line. That is the trade, and on a real dataset it is what makes training possible at all: a little precision on each step in exchange for a hundred times more steps for the same work. Two more things follow from the noise. In Part 9 it turns out the random jostling is actively useful when the landscape has more than one valley. And the never-quite-settling wobble is the reason for the shrinking learning rate mentioned at the end of Part 6: smaller steps late in training calm the jitter down.
Two new hyperparameters have appeared alongside the learning rate, and they are chosen the same way, by trying. Batch size is usually a power of two between 32 and a few thousand, set by how much fits in memory and how much noise you can tolerate. Number of epochs is set by watching the training curve, which is the last thing this part is about.
The training curve
The strip under the demo is the most common chart in all of machine learning, and it is worth pausing on, because it is easy to confuse with the curves from Parts 4 to 6 and it means something different.
| Chart | Horizontal axis | Vertical axis | Shape you expect |
|---|---|---|---|
| Loss curve (Parts 4 to 6) | a weight value | the loss for that weight | a U or a V: high on both sides, lowest at the best weight |
| Loss landscape (Parts 5 and 9) | weight and bias | the loss for that pair | a bowl, or rough country |
| Training curve (this part onwards) | time, in iterations or epochs | the loss the model has right now | falls fast, then flattens out |
The loss curve asks "how good would every possible weight be?" and is drawn once, all at one time. The training curve asks "how is this run going?" and is drawn as it happens, one point per step. It has no U shape because time only runs one way: you do not overshoot the best weight and climb back up the far side of a training curve, you just keep going right. If you saw a training curve rise, that would not be the far side of a U, it would mean the learning rate is too big or something has gone wrong.
Reading one is a skill you now have the pieces for. A curve that plunges and then goes flat has converged, the happy ending from Part 5, and the flat part tells you how many epochs were enough. A curve that is still sloping down when training stops was cut off early. A curve that crawls downhill is a learning rate that is too small; one that climbs or turns into NaN is a learning rate that is too big, exactly the panels from Part 6. A curve that is jagged is mini-batch noise, and the smaller the batch the more jagged it gets. And in the next part the training curve gains a second line, the loss on data the model has not trained on, and the gap between the two lines becomes the most important thing on the chart.
Next: Overfitting and Generalisation: what "good enough" really means, why a model can ace its training data and still fail in the real world, and how held-back data keeps it honest.