Solve It or Search for It

The last part ended on the million dollar question: given a pile of data points, how do we find the weight and bias that make the best predictions?

Before meeting any answers, it is worth feeling the problem in your own hands. Below is a fresh dataset, and your job is to fit it: drag the two sliders to set a weight and a bias, and try to get the line as close to the points as you can. The dashed gaps show how far the line misses each point, and the readout keeps a running score: the average miss across all twelve points.

Two things become obvious after a minute of sliding. First, no straight line passes through all the points. Real data is noisy, measurements wobble, people are inconsistent, sensors drift, so the points scatter around the underlying trend rather than sitting on it. However you set the knobs, some dashed gaps remain. Second, some lines are clearly better than others: the misses shrink as you tune the knobs, and there is one particular weight and bias where the average miss is as small as it can get. That is the goal: not a line that touches every point (no such line exists), but the line whose overall miss is smallest.

There are two broad families of answer. We can solve for the best weight and bias directly with a formula, or we can search for them step by step. Both are worth understanding, because the reason the second one wins is the whole reason this series exists.

Both approaches rest on a single idea we should name before going further: the error. You have already met it: it is the "average miss" you were shrinking in the demo above, a single number measuring how wrong a particular guess at the weight and bias is, in other words, how far the line built from that guess falls from the data points. A good guess has a small error; a poor one has a large error. Whether we solve or search, the goal is identical: find the weight and bias with the smallest error. We will pin "error" down precisely in the next part on loss functions, where it turns out there is more than one reasonable way to keep score; the rough idea is enough for now.

And now the demo's small secret can be told: for that dataset, no straight line, however you set the sliders, can get the error below about 0.33. That number is this problem's floor, the smallest error any line can achieve, and the weight and bias that reach it are the answer to the million dollar question. Scroll back up and hunt for it. The readout turns green when you get there.

Closed-form solutions: solve it in one shot

A closed-form solution is a formula you feed your data into to get the answer straight back, no guessing, no repetition. For fitting a straight line, the best known is the Normal Equation (also called ordinary least squares):

w = (X^T X)^-1 X^T y

Do not worry if the notation looks alien. X is a table of your inputs (one row per data point, plus a column of 1s so the bias is included), and y is the column of targets. X^T means the table flipped on its diagonal, and ^-1 is a kind of "division" for tables of numbers. The important thing is not the symbols, it is that this is one calculation: data goes in, the best weight and bias come out. (You may also run into fancier-sounding routes to the same answer, names like QR decomposition, SVD, or ridge regression. They are refinements for numerical accuracy and stability, not different ideas. File the names away and move on.)

For a single feature, the whole thing collapses to two formulas you can do by hand:

w = Σ (xi - mean(x)) (yi - mean(y))  /  Σ (xi - mean(x))²
b = mean(y) - w × mean(x)

Worked on our study-hours data. Taking the six points from the table in the last part:

Σx = 21      Σy = 404      Σx² = 91      Σxy = 1510      (n = 6)

X^T X = [  6   21 ]       X^T y = [  404 ]
        [ 21   91 ]               [ 1510 ]

det(X^T X) = 6·91 - 21·21 = 105

b = ( 91·404 - 21·1510) / 105 = 5054 / 105 ≈ 48.13
w = (-21·404 +  6·1510) / 105 =  576 / 105 ≈  5.49

So the formula hands us w ≈ 5.49 and b ≈ 48.13, the line y ≈ 5.49x + 48.1, which is exactly the best-fit line drawn through the study-hours scatter in the last part. One calculation, no trial and error.

And it works just as well on the dataset you fitted by hand at the top of this page: feed those twelve points into the formula and it immediately returns w ≈ 0.60 and b ≈ 0.94, the very line you were hunting for with the sliders, found without a single try. That is the appeal of a closed-form solution: your minute of slider-fiddling, replaced by one calculation.

So why not always do this? Three reasons.

Reason 1: it does not scale

The expensive step is inverting X^T X, a matrix whose size is set by the number of features (weights), and building it means passing over all your data. The inversion cost grows with the cube of the feature count: double the features and you roughly octuple the work, and the matrix has to fit in memory. For a single straight line that is nothing. But modern models have thousands, millions, even billions of weights, and inverting a million-by-million matrix is hopeless.

Reason 2: curves make it messy

Real data often follows a curve rather than a line. To bend the model, you add higher powers of the input, each with its own weight:

straight line    y = w·x + b
parabola         y = w2·x² + w1·x + b
cubic            y = w3·x³ + w2·x² + w1·x + b

Here is the part that catches people out: a parabola is plainly not a straight line, so how can a linear method fit one? It comes down to what the word linear is describing. The Normal Equation does not care whether the curve bends; it only cares how the weights combine, and they combine in the simplest way possible, each one multiplied by a number and the results added together. Look again at y = w2·x² + w1·x + b: the weights w2, w1, and b are never squared, never multiplied by each other, never passed through a function. They are only scaled and summed. That is what "linear in the weights" means, and it is all the formula needs.

The way to picture it is to treat x² as a brand-new feature. You already know x for every data point, so you can work out x² for each one too and hand it to the formula as just another column of inputs, sitting beside x. The formula never realises that column is secretly x squared; it just fits a flat relationship across however many input columns you give it (x, x², x³, and so on). So the same one-shot calculation keeps working: you add a column for x², then x³, as far as you want to go.

Try it: bend the line with higher-order terms

The data points below follow an S-shaped curve, and no straight line can fit them. Start with just w1 and b (a straight line) and see how low you can push the error. Then bring in w2 and w3 to bend the curve. Watch the MSE readout: it measures how far the curve is from the points, and it is the "error" we have been talking about.

With only w1 and b, the best you can do is a compromise line through the middle, and the error stays high. Turn w3 up towards 1 and w1 down towards -2 and the curve snaps onto the points. Same fitting machinery, one extra column of inputs.

The catch is that it stops being free. The matrix grows with every term you add, it becomes increasingly delicate to invert, and you are now also guessing the right degree of curve. The neat one-shot formula starts to creak.

Reason 3: sometimes there is no formula at all

The closed-form trick rests on a single move: take the statement "these are the weights that make the error as small as it can possibly be," turn it into an equation, and solve that equation outright for the weights. That "smallest error" is exactly the sweet spot you were hunting for with the sliders at the top of this page: the setting where nudging the weight or the bias in either direction only makes the fit worse. For a straight line the resulting equation is simple enough to solve in one go, which is exactly why the Normal Equation works. But that move is not always available. Push to a genuinely non-linear model, and the equation you would need to solve can simply have no formula-style answer at all.

For the curious: this is a proven dead end, not a "we haven't found it yet." The Abel–Ruffini theorem shows there is no general formula (using only +, -, ×, ÷ and roots) for polynomial equations of degree five or higher. When your best-fit problem reduces to one of those, the closed-form door is mathematically shut. You do not need this theorem for anything else in the series, it is just a satisfying reason why the search-based approach below is not a workaround but a necessity.

And the cousins are no better

It is worth being clear that these three limits are not quirks of the Normal Equation that a smarter formula could dodge. Remember the fancier-sounding relatives mentioned earlier, QR decomposition, SVD, ridge regression: every one of them inherits the same three problems, because they are different routes to the same destination.

Where they do help is in getting a trustworthy answer out of the arithmetic. The Normal Equation's inversion step can misbehave when two of your input columns are nearly identical, say floor area in square metres alongside floor area in square feet, and the numbers it hands back can come out as nonsense. The cousins take a more careful route through the same calculation and stay reliable in those cases. So if you are going to solve in one shot, they are the safer way to do it.

That is the whole of their advantage. The three limits above are untouched:

• Scale. They still work on the whole dataset at once, still need it to fit in memory, and their cost still climbs steeply with the number of weights. SVD is in fact more expensive than a plain inversion, not less. None of them makes a billion-weight model feasible.
• Curves. They fit exactly the same family of models, those that are linear in the weights. Adding x² and x³ columns works, and gets fragile and guessy, in precisely the same way.
• Existence. They all rest on the same underlying move: state the "smallest error" condition as an equation and solve it outright. When that equation has no formula-style answer, having a numerically superior way to solve it does not help, since there is nothing there to solve.

So the closed-form family as a whole, not just its best-known member, hits a wall. And that wall is exactly where the second approach begins.

Gradient descent: search for it step by step

Gradient descent gives up on solving the problem in one shot. Instead it starts with a guess for weight and bias, measures how wrong that guess is, and nudges the values a little in the direction that lowers the error, then repeats, again and again, until the result is good enough.

You have already done this by hand. At the top of this page you moved a slider, watched the error go up or down, and kept moving it the way that made the number smaller. Gradient descent is that, automated: it works out which way to nudge each value, takes a small step that way, and repeats.

Notice the bar: good enough, not perfect. That sounds like a downgrade, but it is exactly what makes the method so powerful. It never inverts a giant matrix, so it scales to billions of weights. And the one question it needs answered at each step is a modest one, "would a slightly larger weight make the error better or worse?", which can be answered for almost any model you can imagine (linear, polynomial, logistic, or a deep neural network). That is why gradient descent keeps working long after every closed-form formula has given up. How it answers that question, and how big a step to take once it has, is what the next few parts of this series are about. And a fit that is merely good enough often generalises better to new data than one tuned to perfection (more on that in Parts 7 and 8). Just like in life, once you settle for good enough, it opens up a whole world of possibilities.

The rest of this series is about gradient descent.

Next: Loss Functions: how to turn "how wrong is this guess?" into a single number, the error we have been leaning on, and why the shape of its curve matters so much to gradient descent.