From Switches to Sums

Part 2 ended with a chip full of switches and nothing else. Here is the question that bothered me for a long time: if all you have is on and off, how do you add 3 and 5? There are no numbers in the machine. There is no "3" anywhere. There is a wire that either carries a voltage or does not. And yet the machine on your desk adds billions of numbers a second and never gets one wrong.

The answer comes in three steps. Agree on a way to write numbers using only on and off. Build a few tiny circuits that combine ons and offs in useful ways. Then wire those circuits together in the same pattern you learned for adding in primary school. By the end of this part you will have watched a working adder built from nothing but gates, and you will understand the one economic fact that explains why AI chips keep using fewer and fewer bits per number.

Counting with two fingers

We write numbers in tens because we have ten fingers. The digits go 0 to 9, and when you run out you carry one to the next column, which is worth ten times more. A machine made of switches has, in effect, two fingers. Its digits go 0 and 1, and when you run out you carry one to the next column, which is worth twice as much. This is binary, and it is the same positional system you already know with a different base.

So the columns of a binary number are worth 1, 2, 4, 8, 16, and so on, doubling each time. The number 5 is one four plus one one, written 0101. The number 3 is 0011. Add them the way you were taught, column by column from the right, carrying when a column overflows, and you get 1000, which is one eight. Three plus five is eight. Each column is a bit, and four of them can hold any number from 0 to 15. Eight bits (a byte) reach 255, and 32 bits reach a little over four billion. A 32-bit number is just 32 wires, each on or off.

Where does the "3" live, then? Nowhere and everywhere. It lives in the agreement that wires two and one are on and the rest are off. The machine does not know it is a three. It only knows which wires are hot. Every number, letter, pixel, and neural network weight in a computer is a convention like this, laid over a pattern of switches.

Four gates

The next step is to build circuits that take some bits in and put a bit out. These are logic gates, and there are only a few, each made from a small handful of transistors.

The simplest is NOT: one input, one output, and the output is the opposite of the input. Two transistors of complementary types wired between the power supply and ground do it. When the input is 1 the lower transistor conducts and pulls the output to 0, and when the input is 0 the upper one conducts and pulls the output to 1. This complementary arrangement is what the C in CMOS (complementary metal-oxide-semiconductor) stands for, and its virtue is that one of the two transistors is always off, so almost no current flows except at the moment of switching. That is the reason chips do not simply melt.

AND takes two inputs and outputs 1 only when both are 1. Put two transistors in series and current only flows if both gates are open. OR outputs 1 when either input is 1. Put two transistors in parallel and current flows if either gate is open. The fourth, XOR (exclusive or), outputs 1 when the inputs are different, and it takes a few more transistors to build. Here they are as tables, since a gate is really nothing more than its table:

That is the entire toolkit. Everything a GPU does, drawing a game, running a language model, is some vast arrangement of these four tables, and any of them can be built from two to twelve transistors. (In practice chip designers build everything from one gate, NAND, which is AND followed by NOT and happens to be the cheapest to make. Any of the others can be assembled from NANDs. You will meet the word again in "NAND flash", the storage in an SSD, which is named after the gate because its memory cells are wired in series the same way.)

Adding one column

Look at the XOR column of the table and then look at the right-hand column of the binary addition from earlier. Adding two bits gives 0 + 0 = 0, 0 + 1 = 1, 1 + 0 = 1, and 1 + 1 = 0 carry 1. The sum bit is exactly A XOR B. And the carry bit, which is 1 only when both inputs are 1, is exactly A AND B. One XOR gate and one AND gate add two bits. This circuit is called a half adder, and it was the moment the whole thing clicked for me: the arithmetic falls out of the tables, nobody had to invent it.

A half adder is not quite enough, because every column except the first also has to take in the carry from the column to its right. A full adder takes three bits in (A, B, and carry-in) and puts two bits out (sum and carry-out). It is two half adders and an OR gate, about 28 transistors in the usual CMOS form. Chain 32 full adders together, each one's carry-out feeding the next one's carry-in, and you have a 32-bit adder that can add any two numbers up to four billion. The one below is a four-bit version you can play with. Click the bits.

There is one practical wrinkle worth knowing about, because it will come back when we talk about clock speed in Part 4. In the chain above, the leftmost adder cannot finish until the carry has rippled through all the ones to its right. For 32 columns that is 32 gate delays in a row, and real adders use cleverer wiring that works out the carries in parallel. The principle is unchanged. It is still XOR, AND, and OR, just arranged so the answer arrives sooner.

Multiplication is repeated addition, laid out flat

Now the first half of our running calculation, w × x. Long multiplication in binary is easier than in decimal, because multiplying by a single binary digit is either "copy the number" (times 1) or "write zeros" (times 0). So multiplying an 8-bit number by another 8-bit number means producing eight rows, each of which is either the top number shifted left or a row of zeros, and then adding the rows up.

Building that in hardware is direct. "Copy the number or write zeros" is an AND gate per bit, so eight rows of eight bits is 64 AND gates. Adding the rows is a grid of full adders. An 8-bit multiplier is therefore a small rectangular array of gates. A 32-bit multiplier is the same pattern with 32 rows of 32, which is 1,024 partial products instead of 64, sixteen times more gates for four times more bits. This is the fact to take away: the size of a multiplier grows with the square of the number of bits. Halve the bits and the multiplier shrinks to roughly a quarter, and gets faster and uses less energy into the bargain.

GPUs do not keep the multiplier and the adder separate. The unit at the heart of every GPU lane is a fused multiply-add (FMA), which computes a × b + c in one step, with one rounding at the end. It is w × x + b exactly, built in silicon, and it is the atom out of which the whole rest of this series is made. When a spec sheet says a card does 82 trillion floating-point operations a second, it is counting FMAs, two operations each, across all the lanes, every clock tick.

Numbers with a decimal point

Whole numbers are not enough for a neural network. Weights are values like 0.0317 and negative 2.4. The convention for those is floating point, and it is the binary version of scientific notation. A floating-point number has three parts: a sign bit, an exponent that says where the point goes, and a mantissa (also called the significand) that holds the actual digits. The name of a format tells you how many bits it spends on each:

The last column is the point. Multiplying two floating-point numbers means multiplying their mantissas and adding their exponents, and the exponent addition is cheap. So the cost of the multiplier is set by the mantissa, and it grows with the square of its width. An FP32 multiplier works on 24-bit mantissas (23 stored plus one implied). An FP8 multiplier works on 4. The FP8 unit is dozens of times smaller, which means you can fit dozens of them in the space of one, running on the same power budget. That table is, in miniature, the story of every Nvidia generation from Volta onward: FP32 gave way to FP16, then BF16, then FP8 on Hopper, then FP4 on Blackwell, and each step bought several times the arithmetic for the same silicon and the same watts. The price is precision, and Part 9 is about why neural networks turn out to tolerate that price remarkably well.

Where this leaves us

You now have the machine's entire vocabulary. Bits are wires. Numbers are agreements about which wires are hot. Gates are handfuls of transistors that implement a four-row table. Adders are chains of gates that follow the primary-school method, multipliers are grids of adders, and a fused multiply-add is w × x + b cast in silicon. The whole of a GPU's arithmetic reduces to that one unit, repeated.

It is worth pausing on how little of the chip the arithmetic actually is. A single FP32 FMA is somewhere in the tens of thousands of transistors. An RTX 4090 has 16,384 of them, so all the lanes together account for something like a billion transistors, give or take. The chip has 76 billion. Where did the other 75 go? Most of them are not doing arithmetic at all. They are storing numbers, moving numbers, and deciding what to do next, and that is the subject of the next two parts. An adder that has nobody to tell it what to add is just a very expensive paperweight.

Next: A Machine That Follows Instructions: registers, the clock, the fetch-decode-execute loop, and a six-instruction program that computes w × x + b on a processor you can single-step.