Gradient descent stops when the model is "good enough." But good enough at what? Here is the uncomfortable truth: a model that predicts its own training data perfectly can still be useless.
The training data is never the point. We already know the targets for those rows, that is what made them training data. The entire reason for building a model is to make predictions for inputs it has never seen: the next student, the next house, tomorrow's sensor reading. How well a model performs on unseen data is called generalisation, and it is the only score that ultimately matters.
Memorising versus learning
Think of two students preparing for an exam.
The first student works through past papers to understand the method: why each step follows from the last, how to recognise which technique a question calls for. The second student memorises the past papers, every question, every answer, word for word. Quiz them both on last year's paper and the memoriser scores 100%, beating the honest learner. Put a new paper in front of them and the memoriser collapses, while the learner does about as well as they always did.
A machine learning model can be either student, and nothing in the training loss tells you which one you have. A model that has memorised its training data, noise, outliers, quirks and all, is called overfitted. It has fitted itself not just to the underlying pattern but to the random scatter around it, and random scatter does not repeat in new data.
There is an opposite failure, too: a model too simple to capture the pattern at all (imagine forcing a flat horizontal line through clearly rising data). That is underfitting. It does badly everywhere, on training and new data alike, which at least has the virtue of being easy to notice.
See it happen
Remember from Part 3 that adding higher powers of x gives a model more knobs and lets it bend. More knobs means more capacity, and more capacity is exactly what a memoriser needs.
Below, eleven data points (dark dots) were used to fit three models: a straight line (2 knobs), a gentle cubic (4 knobs), and a degree-9 polynomial (10 knobs, almost one per data point). Five more points from the same source were held back and never shown to the models. Fit each model, admire its training score, then reveal the held-back points. And this time the knobs themselves are exposed below the chart: every coefficient arrives set to its fitted value, and you can grab any of them and see what your meddling does to both scores.
The model's knobs (set to the fitted values — drag any of them):
Watch the two numbers as you switch models. The straight line has a modest training error and, once revealed, a similar held-back error: it learned the trend. The degree-9 polynomial drives its training error down close to zero, it snakes through nearly every training point, but between and beyond those points it swings wildly, and its held-back error explodes. That gap between "brilliant on training data" and "terrible on new data" is overfitting. The cubic sits in between: a little more flexible than the line, no real gain, because the underlying pattern here really is close to straight.
Now play with the knobs. Nudge the straight line's w1 and the curve tilts politely, the training MSE creeping up as you stray from the fitted value; you could almost re-fit it by hand. Then switch to degree 9 and breathe on w9, a knob whose fitted value is 0.0003, and the curve convulses. That fragility is what 10 delicately counterbalanced knobs look like: enormous coefficients cancelling each other to the fourth decimal place just to thread through eleven training points. A model that learned the trend barely notices a nudged knob; a model that memorised the points falls apart at a touch.
The fix: hold data back
The memorising student was only exposed because we handed them a paper they had not seen. Models are policed the same way. Before training starts, the dataset is split, a common recipe being:
- Training set (say 70–80%): the only data gradient descent ever sees. Weights and biases are fitted to this and nothing else.
- Validation set (say 10–15%): kept aside during training but checked regularly, like mock exams. If training loss keeps falling while validation loss starts rising, the model has begun memorising, and we stop. This is the early stopping rule from Part 5, now with its full explanation. The validation set also referees choices like the learning rate or how many
x²,x³terms to allow. - Test set (say 10–15%): touched exactly once, at the very end, for the final grade. It has had no influence on any decision, so it is the closest thing we have to genuinely unseen data.
Only one of them changes the model
There is a sharp line running through that list, and it is the most important thing to understand about the three sets: only the training set ever changes the weights and biases.
Gradient descent runs on training data alone. Predict, measure the loss, compute the gradients, nudge every knob downhill, repeat. That loop, the entire mechanism of the last three parts, only ever touches training rows.
Validation and test data are put through the model too, but the process stops halfway. Predictions are made, the loss is computed, and then… nothing. No gradients, no nudges, no updates. The weights and biases sit exactly as they were before, unchanged, because the point is to observe the model, not to improve it. It is the difference between practising and being examined: the same questions, the same effort, but during the exam nobody corrects you mid-answer.
| Model makes predictions | Loss is measured | Weights and biases updated | |
|---|---|---|---|
| Training set | yes | yes | yes |
| Validation set | yes | yes | no |
| Test set | yes | yes | no |
This is also what makes the whole scheme honest. If validation data updated the weights, the model would have learned from it, and its validation score would be just another training score, telling us nothing new. The measurement only means something as long as the model stays frozen while we take it.
One caveat, and it is the reason validation and test are separate sets. Validation data never touches the weights directly, but you look at the result and act on it, stopping training here, choosing a different learning rate there. The knobs are not being tuned by gradient descent, they are being tuned by you, slowly, through a hundred small decisions. That is a subtler kind of learning, and it is why the final verdict has to come from a test set that has influenced nothing at all.
Where benchmark scores come from
This is also the answer to something you have almost certainly seen without knowing what you were looking at. Every time a lab releases a model, the announcement comes with a table of scores: 88.7% on this, 71.4% on that, a chart with competitors' bars next to it. Those are test set results.
The analogy holds all the way through. The student revises from past papers (training set), takes mock exams to see how it is going (validation set), and then sits the real exam once (test set) — and it is that last number, not their revision, that goes on the certificate for everyone else to read. Benchmarks are the model's exam results, published.
What makes them useful is precisely the discipline described above. The questions are held back from training, the model sees them once, and the score is reported. Because the exam papers are standardised and shared across the industry (names like MMLU, GSM8K, and HumanEval are just widely-agreed question sets, on general knowledge, grade-school maths, and programming respectively), everyone's model can sit the same exam and the numbers can be compared.
And the same discipline explains how benchmarks go wrong. Two failures are worth recognising:
- Contamination. These test sets are published on the internet so that others can use them. Which means they can quietly end up inside the next model's training data, scraped along with everything else. A model that trained on the exam paper will ace the exam, and the score means nothing. It is the student who found the answer key, and this is a genuine and recurring problem in the field, not a hypothetical one.
- Teaching to the test. If a lab tunes its choices to push a benchmark number up, run after run, that benchmark has quietly become a validation set, consulted many times and leaking its quirks into the decisions. The published number drifts away from what it claims to measure.
None of which makes benchmarks useless, they are the best comparison tool available. But it does explain the healthy scepticism you see whenever a new score is announced, and why "how did it do on your problem?" is always the better question. A model's exam results, like a student's, tell you something real, and never the whole story.
Key takeaways
A model's true score is measured on data it never trained on: that is generalisation. When training performance is excellent but unseen-data performance is poor, the model has overfitted, and the more knobs a model has relative to its data, the easier memorisation becomes. Splitting data into training, validation, and test sets is how practitioners catch it, and early stopping is that split acting as a tripwire during training itself.
Keep this idea close, because the next part hinges on it: it turns out that failing to find the perfect minimum of the loss can be a blessing in disguise.
Next: Local vs Global Minima: what happens when the loss surface has hills and valleys, and why ending up at a local minimum is not always a bad thing.