Local vs Global Minima

In Part 5 the loss stopped being a curve and became a surface, and it had a very convenient shape: a smooth bowl with one lowest point, so rolling downhill from anywhere arrived at the same destination.

Since this part is all about the shape of that surface, one piece of vocabulary needs pinning down first, because it is easy to mix up with something else you have seen.

What the "landscape" is a picture of

From here on you will see the loss surface called a loss landscape. It is worth being precise about what that landscape is a map of, because it is not a picture of your data, and it is not the shape of the line your model draws.

Think of it as a map of every possible version of your model:

• The ground is all the settings you could choose. Every position on it is one specific combination of weight and bias, one candidate model. Walking east means a larger weight; walking north, a larger bias.
• The height at each position is that candidate's loss, its single score for how badly it fits the training data. High ground means a poor model, low ground means a good one.
• Your current model is a single point standing somewhere on that ground, and training is that point walking downhill.

So the landscape has nothing to do with whether your fitted line is straight or curved, and nothing to do with how the data is scattered. It is a chart of how wrong every possible setting would be, laid out so that "better" means "lower". Two things that are easy to confuse, side by side:

Everything below is about the second picture only.

The bowl was the friendly case

Here is that landscape again, the one Part 5 left us with, now that you know exactly what its axes mean: the two knobs on the floor, the loss as height.

Notice what that shape promises. There is exactly one low point, and every part of the surface slopes towards it. A ball placed anywhere on this bowl, released, rolls to the same spot. Starting position does not matter, the route does not matter, and "keep going downhill" is a complete strategy: it always finds the best possible weight and bias.

That smooth bowl was not luck, and it was not general. It is a gift that comes specifically from fitting a straight line with MSE, and this part is about what happens when the gift is withdrawn. For almost any model more interesting than a line, the landscape is not a bowl at all: it is rough country, with several separate valleys of different depths divided by higher ground.

Why the bowl was a special case

It is worth being clear about what was doing the work in that tidy picture. Two things had to be true at once: the model was a straight line, and the loss was MSE. Squared errors grow smoothly and symmetrically as a knob moves away from its best value, and with a straight line every knob affects the prediction in the simplest possible way. Put those together and the landscape can only come out as a single smooth bowl, however many knobs you add.

Break either condition and the guarantee evaporates. Add higher-order terms, stack layers of neurons, pass values through the activation functions of Part 9, and the same weight can now influence the output through several routes at once, sometimes reinforcing, sometimes cancelling. The landscape stops being a bowl and develops multiple hills and valleys: several separate low spots, of different depths, separated by higher ground.

Nothing about gradient descent changes. It still stands at a point, reads the local slope, and steps downhill. The difference is what that now guarantees, which is much less than it did before.

Local minima and global minima

Picture that landscape as rough country with many valleys, remembering that a position on it is a setting of the knobs, not a data point. Gradient descent is a ball rolling downhill from wherever it is dropped:

• It will always find the nearest valley bottom, the local minimum.
• It may not find the lowest point on the entire surface, the global minimum.

Where the ball is dropped decides which valley it ends up in. And where it is dropped is just the initialisation: the values the weight and bias are given before training starts. Recall from Part 3 that nobody knows the right values in advance, so training begins from a guess, usually random numbers. That guess is the starting position on the landscape. Two runs of the same training on the same data, differing only in their random starting numbers, can therefore roll into different valleys and end up as genuinely different models.

Is a local minimum bad?

Counterintuitively, the answer is often no.

Researchers studying deep neural networks found something surprising: the global minimum is frequently worse than a good local minimum in practice. A paper by Choromanska et al. (The Loss Surfaces of Multilayer Networks) put it this way:

"...we prove that recovering the global minimum becomes harder as the network size increases and that it is in practice irrelevant as global minimum often leads to overfitting."

This should ring a bell from the previous part: overfitting means the model fits the training data too perfectly, memorising it rather than learning general patterns. The global minimum is, by definition, the set of parameter values most precisely matched to the training set, noise, outliers, and all, so it is exactly where memorisation is at its worst.

A local minimum, by contrast, tends to capture the broader trends in the data without over-specialising. It generalises better to unseen inputs.

Interactive demo: the landscape and where you start

First, how to read the picture

The demo below draws the same kind of landscape, but seen from directly above, the way a golf course looks from a drone hovering over it. You lose the sense of depth, so height has to be shown some other way: with colour.

• Left to right is the weight; bottom to top is the bias. Every pixel is one setting of the two knobs, one candidate model.
• Colour is height, meaning loss. Warm colours are high ground and cold colours are low ground:
  • Red / orange — high loss, the hilltops. Models here fit the data badly.
  • Green / teal — the middle slopes.
  • Dark blue — low loss, the valley floors. The good models live here.
• The white dot is your starting model, and the white trail is the path gradient descent takes as it walks downhill.

That is the whole trick of this kind of chart: it flattens a three-dimensional surface onto the page and uses colour where the third dimension used to be. The bowl from earlier, viewed this way, would be a single blue pool in the middle with red edges all around it. The landscape below is not so tidy: look for several separate blue pools, some deeper in colour than others, with ridges of green and red between them. Those pools are the valleys, and a ball can only ever roll into the one it starts nearest to.

Try it

The white dot is where gradient descent starts, which in a real training run would be set by the random initialisation. Here you get to place it yourself: click anywhere on the map, press Descend from here to watch the model roll to the nearest valley, then start it somewhere else and run again. The same model, the same data, the same algorithm, and a different answer depending only on where it began.

What to explore

Click around different parts of the map and run the descent each time:

• Start on red high ground and watch the path run downhill fast at first, the steep part, then slow as the colour cools and the ground levels out.
• Start on opposite sides of a green ridge. Two starting points that look close together can end in completely different blue pools, because the ridge between them is a watershed: gradient descent will not climb over it.
• Compare where the runs stop. Some settle in a paler blue pool, others in a darker one. All of them are valleys, all of them are places where every direction is uphill, but they are not equally good. Only the darkest is the lowest.

That last point is the one to take away, and it has a name.

The key takeaway

Because gradient descent follows the local slope, the result depends on where you start. For complex models this is unavoidable. The practical response is:

• Random restarts: train several times from different random starting values (different initialisations) and keep the best result.
• Accept local minima: for large neural networks, local minima are often good enough and generalise better than the global minimum.
• Batch training tricks (mini-batch SGD): randomly sampling batches of data introduces noise into the gradient, which helps the optimiser escape shallow local minima.

Beyond one feature: multivariate regression

One last piece of context before we leave plain linear regression behind. Throughout this series the target has depended on a single feature, one input x with one weight. Most real problems are not so tidy. Predicting a house price might draw on floor area, number of bedrooms, age, and location all at once. This is multivariate regression: several features, each with its own weight.

The model simply grows more terms, one weight per feature, plus the single shared bias:

y = w1·x1 + w2·x2 + w3·x3 + ... + b

For instance, to predict a house price from three features, its floor area, number of bedrooms, and age, the data might look like this:

Training finds one weight for each feature, plus a bias. For this data a fitted model comes out as:

price = 3 × area + 25 × bedrooms - 2 × age + 50

Each weight is a price sensitivity you can read straight off: about £3k for every extra square metre, £25k per bedroom, and £2k off for every year of age, on top of a £50k base. Feed a new house's three numbers into that equation and out comes its predicted price.

Nothing else about the recipe changes. You still make a prediction, measure the loss, and take a partial derivative with respect to every weight (and the bias) to see which way each one should move, then step them all downhill together. The only thing that grows is the number of dimensions: the loss surface now lives in a space with one axis per weight, far too many to draw. But gradient descent never needed the picture. It only ever needed the slope in each direction, and that it can always compute.

That is the engine behind everything from linear regression to the largest neural networks: measure how wrong you are, then roll downhill. What changes from here is not the engine but the shape of the model it drives, and that is where the next part comes in.

Next: Activation Functions: the trick that lets a model bend beyond straight lines to fit curves and other non-linear shapes.