Weights, Bias, and the Linear Model

The fundamental equation of linear regression is:

y = weight × x + bias

This is just the equation of a straight line, where weight is the slope and bias is the y-intercept. But understanding exactly what each knob does is the foundation of understanding machine learning, because everything in gradient descent is about finding the right values for these two numbers.

What weight does

Weight controls the angle of the line. Think of placing a physical weight on one end of a seesaw: the heavier the weight, the steeper the tilt.

• A weight of 0 makes the line completely flat (horizontal).
• A positive weight makes the line slope upward as x increases.
• A negative weight makes the line slope downward.
• A larger absolute value makes the slope steeper.

What bias does

Bias controls the vertical position of the line, where it sits when x is zero. It is the y-intercept.

• If weight is zero and you only change bias, the line moves straight up and down.
• Bias tells you "how far away from neutral is my starting point."

The name "bias" is fitting: it biases the output up or down regardless of the input.

Explore weight and bias

Use the sliders below to change the weight and bias and see how the line responds. Notice:

• When weight is 0, the line is flat. Change bias to move it up or down.
• The red dot marks the intercept point (where the line crosses x = 0), which always equals the bias.
• As you increase weight, the line tilts. At weight = 1 and bias = 0, y = x (a 45° diagonal).

Any line, two numbers

Here is the key insight: with just these two numbers, you can represent any straight line. Once you know the weight and bias, you can reproduce the line exactly.

This is worth pausing on, because it is the hinge the whole subject turns on. In Part 1 the regression line was a picture: a stroke drawn through a cloud of dots, and you made a prediction by tracing it with your eye. Weight and bias are that same line written as numbers. The picture and the pair of numbers are two views of one thing, and you can move between them freely:

Trading the picture for the numbers is what makes machine learning possible, because numbers are something a computer can work with in ways it could never work with a drawing. You can put them in a formula and get a prediction without plotting anything:

prediction = weight × new_input + bias

You can compare two candidate lines by comparing their numbers. You can measure exactly how wrong a line is, and, as the rest of this series is about, you can adjust those two numbers a nudge at a time until the line fits better. None of that is possible while the line is a stroke of ink; all of it is routine once the line is a pair of numbers. Everything from here on, loss functions, gradient descent, entire neural networks, is arithmetic on numbers like these. A modern language model is the same idea with a few hundred billion of them instead of two.

For Galton's data the weight was approximately 0.66 and the bias 23, giving the model:

predicted_child_height = 0.66 × parent_mid_height + 23

Feed in a parent mid-height of 71 inches and you get a predicted child height of about 69.9 inches, even for a family not in the original dataset.

Now the million dollar question

Notice what we have quietly been given so far. Galton did the hard part for us: he worked out that the weight was 0.66 and the bias 23, and we have simply been using his answer. The demo above let you see what those two numbers mean, and the formula lets you use them once you have them. Neither tells you where they came from.

That is the real problem. In practice you are handed a dataset of (feature, target) pairs and nothing else, no Victorian scientist attached, and two questions stare back at you:

• How do you work out the weight and bias for any dataset? Galton had a few hundred families and years to think. A real dataset might have millions of rows, and nobody is going to eyeball a line through those.
• How do you know the values you found are the best ones? Suppose you settle on a weight of 0.7. Would 0.68 have fitted better? Would 0.9? "It looks about right" is not an answer a computer can give, or check.

For example, suppose you recorded how many hours each student in a class studied and the exam score they went on to get:

Each row is one (feature, target) pair: the feature is the hours studied, the target is the score. Plotted, these points do not sit perfectly on a straight line (real data never does), but they clearly trend upward. This is what a dataset actually looks like, and here it is only six rows; a real one might have hundreds, thousands, or even millions of them.

The whole idea of machine learning is this: given a pile of data points like these, find the single weight and bias whose line passes as close as possible to all of them at once. That line becomes your trained model. Feed it a number of study hours it has never seen and it predicts a score.

The million dollar question in machine learning is: how do we find the values for weight and bias that make the best predictions on our data?

There are two broad families of answer. We can solve for the best weight and bias directly with a formula, or we can search for them step by step. Both are worth understanding, because the reason the second one wins is the whole reason this series exists, and that face-off is exactly what the next part is about.

Next: Solve It or Search for It: the two ways to find the best weight and bias, why the one-shot formula runs out of road, and why searching step by step powers all of modern machine learning.