In the previous part we kept using the word error: how far a particular weight and bias land from a good fit. To do anything useful with that idea, we first need to turn it into a number we can actually compute. That number is called the loss (or cost), and the formula that produces it is the loss function.
Keep one thing front of mind for this whole part, because it is easy to lose sight of: the loss is a score for a specific pair of numbers, one weight and one bias. Change the weight, and you get a different loss. Change the bias, and you get a different loss again. Loss is not a property of your data, and not a property of "the model" in some vague sense, it is the answer to a very precise question: how badly does this particular guess at the weight and bias fit my data?
From predictions to a single number
So let us score one. Take the guess weight = 4, bias = 45, which describes the line y = 4 × hours + 45. It came from nowhere in particular, which is exactly the situation you are in before training. The question of this section is: how bad is that particular guess?, and the answer we want is a single number.
To work it out we need data to judge it against, the same study-hours data from Part 2, where each row is a feature (hours studied) paired with its target (the exam score that student actually got):
| Hours studied (feature) | Exam score (target) |
|---|---|
| 1 | 53 |
| 2 | 60 |
| 3 | 64 |
| 4 | 71 |
| 5 | 75 |
| 6 | 81 |
The trick a loss function uses is to turn this data into an exam for our guess, by pretending we do not know the targets. Picture covering up the score column, so all we are left with is the features and a set of blanks for our line to fill in:
| Hours studied (feature) | Exam score (target) |
|---|---|
| 1 | ? |
| 2 | ? |
| 3 | ? |
| 4 | ? |
| 5 | ? |
| 6 | ? |
Now we let the guess sit the exam. Every feature goes through y = 4 × hours + 45 to produce a predicted score. Then we uncover the real targets and set them beside those predictions to see how far off each one is:
| Hours (x) | Our prediction (4x + 45) |
Actual target | Error (prediction - actual) |
|---|---|---|---|
| 1 | 49 | 53 | -4 |
| 2 | 53 | 60 | -7 |
| 3 | 57 | 64 | -7 |
| 4 | 61 | 71 | -10 |
| 5 | 65 | 75 | -10 |
| 6 | 69 | 81 | -12 |
Even by eye the fit is poor: the line runs below almost every point, and the gap grows the longer the study time. But "clearly off" is not something an algorithm can act on. To improve the line we first have to measure exactly how wrong it is. The value in that last column, error = prediction - actual, is the prediction error for a single data point.
That leaves us with one error per data point: six numbers, all belonging to this one guess of weight 4 and bias 45. Useful, but six numbers cannot be compared against another guess's six numbers at a glance, and a computer cannot act on them.
And six is the friendly case. A real dataset has thousands or millions of rows, so this guess would produce millions of individual errors. Nobody can read that. Knowing that row 407,912 was off by 3.2 tells you nothing on its own, it is a single grain of sand, and scrolling through a million of them will not tell you whether the guess is any good, let alone whether it beats the guess you tried before. Individual errors do not add up to a judgement, no matter how many of them you have.
What we want is to boil them all down to a single number, one score for w = 4, b = 45, that behaves sensibly whether it came from six rows or six million. That is exactly what a loss function does, and the only real question is how to do the boiling down.
First, why not just average the errors?
The obvious move is to add the six errors up and divide by six. Try it: (-4 - 7 - 7 - 10 - 10 - 12) / 6 = -8.33. That works here only by luck, because this line happens to sit below every point, so all six errors are negative.
Now imagine a line that runs through the middle of the data and misses one point by +20 and another by -20. Those two cancel, contributing zero to the average, and a badly-off line scores as if it were flawless. A useful score has to treat "20 too high" and "20 too low" as equally wrong, which means getting rid of the minus signs before averaging. The two standard loss functions are simply the two obvious ways to do that: square them, or take their absolute value.
Both of the formulas below are written in the standard notation you will meet everywhere else, so it is worth two lines to decode it:
| Symbol | Meaning |
|---|---|
yᵢ |
the actual target of data point i (the score the student really got) |
ŷᵢ |
the predicted value for that point, "y-hat", what our line says |
n |
how many data points there are |
Σ from i=1 to n |
"add this up for every data point, from the first to the last" |
Mean Squared Error (MSE)
Square each error, then average. Squaring kills the minus signs (a negative times a negative is positive) and, as a side effect, penalises large errors more heavily: being off by 10 contributes 100, while being off by 2 contributes only 4.
Run it on our guess of w = 4, b = 45, using the same six errors from the table above:
| Hours (x) | Error | Error² |
|---|---|---|
| 1 | -4 | 16 |
| 2 | -7 | 49 |
| 3 | -7 | 49 |
| 4 | -10 | 100 |
| 5 | -10 | 100 |
| 6 | -12 | 144 |
| total | 458 |
MSE = 458 / 6 = 76.33
76.33 is the loss for weight 4 and bias 45. One number, standing in for the whole table. Notice the last row contributes 144 of the 458, nearly a third of the total from a single student, because squaring makes the worst-fitted points shout loudest.
One quirk worth knowing: since we squared exam-score points, the 76.33 is in points squared, so it is not directly comparable to anything in the data. It is a score for ranking guesses, not a quantity with a natural meaning. (Take its square root and you get 8.74, which is in points, and is what people mean by RMSE.)
Mean Absolute Error (MAE)
Take the absolute value of each error, the size with the minus sign thrown away, then average. Every error contributes in proportion to its size, with no extra punishment for the large ones.
The same guess, the same six errors:
| Hours (x) | Error | Error size |
|---|---|---|
| 1 | -4 | 4 |
| 2 | -7 | 7 |
| 3 | -7 | 7 |
| 4 | -10 | 10 |
| 5 | -10 | 10 |
| 6 | -12 | 12 |
| total | 50 |
MAE = 50 / 6 = 8.33
8.33 is the same guess's loss under MAE, and this one is plainly readable: on average, the line y = 4x + 45 misses a student's score by about 8.3 points. That interpretability is MAE's great charm.
Two scores, one guess, both perfectly valid. They disagree on emphasis rather than on facts: MSE's 76.33 was dominated by the worst point, while MAE's 8.33 gives that student exactly the same say as everyone else. Which you should prefer depends on whether a few big misses are a catastrophe or just life, and the last section of this part lets you watch the two scores disagree about exactly that. What matters right now is the shape of what we just did: pick a weight and bias → predict → measure the misses → boil them down to one number. Do that with any formula you like, and you have a loss function.
Trying different weights
To see the loss earn its keep, watch what happens to it as we change the weight. For this we will switch to a clean dataset whose perfect line is exactly y = 2x + 1, and to keep things simple we will hold the bias fixed at 1 and only move the weight. (A real model tunes the bias as well, but freezing it lets us watch one number change at a time.)
Start with a weight that is clearly too small, weight = 1. We run each feature through y = 1·x + 1, compare with the actual target, and note what each row contributes to MAE (the size of the error) and to MSE (the error squared):
| x | Actual | Prediction (1·x + 1) |
Error | Error size (MAE) | Error² (MSE) |
|---|---|---|---|---|---|
| 1 | 3 | 2 | -1 | 1 | 1 |
| 2 | 5 | 3 | -2 | 2 | 4 |
| 3 | 7 | 4 | -3 | 3 | 9 |
| 4 | 9 | 5 | -4 | 4 | 16 |
Averaging the last two columns gives the two loss scores for this weight:
MAE = (1 + 2 + 3 + 4) / 4 = 2.5
MSE = (1 + 4 + 9 + 16) / 4 = 7.5
Both are large, because the line is far too shallow.
Now nudge the weight up to weight = 1.5 and redo the same table:
| x | Actual | Prediction (1.5·x + 1) |
Error | Error size (MAE) | Error² (MSE) |
|---|---|---|---|---|---|
| 1 | 3 | 2.5 | -0.5 | 0.5 | 0.25 |
| 2 | 5 | 4 | -1 | 1 | 1 |
| 3 | 7 | 5.5 | -1.5 | 1.5 | 2.25 |
| 4 | 9 | 7 | -2 | 2 | 4 |
MAE = (0.5 + 1 + 1.5 + 2) / 4 = 1.25
MSE = (0.25 + 1 + 2.25 + 4) / 4 = 1.875
Both scores dropped (MAE from 2.5 to 1.25, MSE from 7.5 to 1.875), so weight 1.5 fits better than weight 1.
If we keep repeating this for a whole range of weights, a clear pattern emerges:
| Weight (bias fixed at 1) | MAE | MSE |
|---|---|---|
| 0 | 5 | 30 |
| 0.5 | 3.75 | 16.875 |
| 1 | 2.5 | 7.5 |
| 1.5 | 1.25 | 1.875 |
| 2 | 0 | 0 |
| 2.5 | 1.25 | 1.875 |
| 3 | 2.5 | 7.5 |
| 3.5 | 3.75 | 16.875 |
| 4 | 5 | 30 |
The loss falls as the weight climbs toward 2, touches zero exactly at the ideal weight (where the line matches the data perfectly), then climbs again as we overshoot. The closer the weight gets to its ideal value, the smaller the error. Finding that ideal weight is the whole game.
Try it yourself
Drag the slider to change the weight, with the bias fixed at 1. The line tilts, the gap at each point grows or shrinks, and the MAE and MSE at the bottom of the graph update as you go. See if you can settle on the weight that drives both losses to zero.
Loss curve
A single loss score only judges one guess. A loss curve tracks the loss as the weight changes: it sweeps through a whole range of weights and records how large the loss is at each one, turning a pile of separate scores into a single shape we can read at a glance.
That shape is what points us to the answer. Where the curve runs high, the weight there is a poor choice and the loss is large; where it dips, the weight fits better. The single lowest point of the curve marks the weight that gives the smallest loss of all, and that is almost certainly the best value to choose for our line. Finding the bottom of this curve is what the rest of training comes down to.
Plotting the MAE and MSE columns against the weight turns that table into a pair of curves:
Why the shape of the loss curve matters
The two curves above are not just different sizes, they are different shapes, and that shape turns out to be critical for gradient descent.
MSE gives a smooth U-shaped (parabolic) curve. Because errors are squared, the loss increases smoothly and symmetrically away from the minimum. Crucially, the curve has a well-defined gradient (slope) everywhere, steeper far from the minimum, nearly flat at the bottom. This gradient is exactly what gradient descent uses to know which direction to move and how far.
MAE gives a V-shaped curve. The slopes on either side are constant. There is no "get steeper as you move further away" behaviour. The gradient gives less information about how far you are from the minimum.
This is why MSE is the more commonly used loss function for regression: its smooth parabolic shape gives gradient descent a clear signal to follow all the way to the bottom.
Interactive demo: the loss curve
The animation below shows what happens as the weight changes from 0 to 4, with bias fixed at 1 (true line: y = 2x + 1). Watch how the predicted line (red) moves relative to the data, and how the MSE and MAE loss curves build up. The ideal weight is 2, that is the bottom of both curves.
Press Start to run the animation. Press Pause to freeze it at any point.
Key observations
- The minimum of both curves occurs at weight = 2, the true weight of the data-generating process (
y = 2x + 1). - The MSE curve is a smooth parabola. Far from the minimum the slope is steep; close to the minimum it flattens. This gives gradient descent a signal that scales with distance.
- The MAE curve has constant slopes on each side, and at the very bottom the two sides meet in a sharp corner. Right at that corner the slope jumps from one value to the other with nothing in between, so there is no gentle "you are nearly there" signal, exactly where precision matters most.
- Squaring errors in MSE means large outliers get penalised heavily. This is both a feature (outliers matter a lot) and a bug (outliers can dominate the training). The demo below shows this trade-off in action.
Outliers: where MSE and MAE disagree
Real datasets contain the occasional rogue point: a sensor glitch, a typo, the student who partied the night before the exam. Because MSE squares every error, one bad point can contribute more to the loss than all the good points combined, and the best-fit weight gets dragged towards it. MAE treats every unit of error equally, so a single outlier barely moves it.
Tick the box below to add one outlier, a student who studied 4 hours but scored as if they had studied none, and watch what happens to the two loss curves. The marker on each curve shows the weight at its lowest point.
Without the outlier, both losses agree: the best weight is 2. Add the outlier and the MSE minimum lurches away from 2, because that one squared error of around 49 outweighs everything else, while the MAE minimum barely moves. Neither behaviour is "right": if your outliers are genuine, rare-but-real events, you may want the model to respect them; if they are glitches, MAE's stubbornness is a virtue. Knowing which loss you are optimising, and what it does with your worst data points, is part of understanding what your model will learn.
Next: Gradient Descent: how to read the slope of this loss curve and use it to step the weight and bias toward the bottom, one iteration at a time.