Here is a comparison that confused me for years. In 2022 you could buy a top-end desktop CPU with 16 cores, and an RTX 4090 with 16,384. Same year, same class of manufacturing process, similar transistor budgets, a thousand-fold difference in the headline number. Either the GPU is a thousand times faster, which it is not, or "core" means two different things, which it does.
Part 4 and Part 5 already contain the explanation. A CPU core is a fetch-decode-execute loop wrapped in an enormous apparatus for making one instruction stream never wait: deep pipelines, out-of-order execution, branch predictors, large caches. Most of the core is that apparatus, and the arithmetic is a small island in the middle. What Nvidia calls a CUDA core is, roughly, the island on its own: one fused multiply-add lane and a slice of register file, with the fetching, decoding, and scheduling shared among 32 of them. A CPU core with its private caches runs to a few hundred million transistors. Divide the AD102's 76 billion by its 16,384 lanes and you get under five million per lane, everything included.
So the question is not which is faster. It is what each one is fast at, and this part is about making that visible.
Latency and throughput, again
The distinction from Part 5 comes straight back, now applied to processors rather than memory. A CPU is a latency-oriented design. It is built to finish one task in the least possible time. A GPU is a throughput-oriented design. It is built to finish the most tasks per second, and it is entirely relaxed about how long any one of them takes.
The analogy I use is a Formula One car against a fleet of buses. If you need to get one person across town, the car wins by a mile. If you need to move ten thousand people, the buses win, even though every individual passenger has a slower, more crowded trip. Nobody thinks the buses are "faster than" the car. They solve a different problem. The analogy breaks in one place worth naming: buses are still usable for one passenger, just wastefully, whereas a GPU given a single sequential task is not merely wasteful but actively slower than a CPU, as the race below will show.
The shape of the work
What decides which design wins is the shape of the job, and there is really only one question to ask: how much of the work is independent?
Colouring the pixels of a frame is the extreme case. Each of the eight million pixels on a 4K screen gets its colour from the same calculation applied to its own inputs, and no pixel needs to know the answer for any other. All eight million could be done at once if you had eight million lanes. Work like this is called embarrassingly parallel, and graphics is why GPUs exist.
Now look at the running calculation of this series with the same eye. A layer of a neural network takes a vector of inputs and produces a vector of outputs, and every output is its own w × x + b summed over the inputs. ML Basics built one neuron at a time. A real layer has thousands of them, and every one is independent of the others: same calculation, different weights, no communication needed until the layer is done. Multiplying matrices, which is what a transformer spends nearly all its time doing, has the same property, since each element of the result is its own dot product. A neural network is, at the level that matters, an enormous number of identical independent multiply-adds. It has the shape of a picture.
Then there is the other kind of work. Parsing a sentence, where each word's meaning depends on what came before it. Walking a linked list, where you cannot find the next item until you have read the current one. A program full of if statements, where the next instruction depends on the last result. This work is a chain, and a chain cannot be spread across lanes because each link waits for the previous one. On a chain, the only thing that matters is how fast one lane can go, and that is the CPU's whole specialty.
Gene Amdahl put a number on this in a 1967 paper. If some fraction of a job is inherently sequential, that fraction caps the speedup no matter how many lanes you add. If a tenth of the work is a chain, then even with infinitely many lanes you can go at most ten times faster, because the chain still takes a tenth of the original time. This is Amdahl's law, and it is why real systems contain both kinds of processor: the GPU eats the wide part, and the CPU is kept for the chain.
The race
The demo below pits a plausible 2022 desktop CPU against an RTX 4090 on a job made of N independent items, each of which is a chain of L multiply-adds that must happen in order. It counts arithmetic only, not memory, so it flatters both machines, but the shape of the result is right. The CPU is modelled as 16 cores each doing 16 multiply-adds per tick at 5 GHz, with 4 ticks of latency per operation. The GPU is 16,384 lanes at 2.5 GHz, also with 4 ticks of latency. Drag the sliders or try the presets.
Play with it and a pattern appears. With one long chain the CPU wins, because the GPU's slower clock is all that matters and 16,383 of its lanes have nothing to do. As soon as there are a few thousand independent items the GPU pulls ahead, and by the time there are millions it is dozens of times faster. In this model the crossover comes at a couple of thousand items, once the CPU's 256 lanes are saturated and the GPU's slower clock stops mattering, and the GPU does not reach full speed until there are more items than it has lanes. Below that, most of it is idle.
The real gap is bigger than the demo shows, in both directions. On the chain, a real CPU would also be using its branch predictor and out-of-order execution to squeeze out more, and a real GPU has other overheads. On the wide job, the CPU would run out of memory bandwidth long before the GPU did. But the crossover, and the reason for it, is faithful.
One instruction, many lanes
There is a detail in how the GPU achieves its width that explains a great deal of what comes later. It does not have 16,384 copies of the clerk from Part 4. It has a much smaller number of clerks, each of whom reads an instruction aloud once and has 32 calculators execute it simultaneously, each on its own numbers. One fetch, one decode, 32 executes. The fetching and decoding machinery, which is the expensive part of a core, is amortised over 32 lanes.
The general name for this is SIMD, single instruction, multiple data, and CPUs do it too on a smaller scale: a modern x86 core has instructions that operate on 8 or 16 floating-point numbers at once, which is where the "16 multiply-adds per tick" in the demo came from. The GPU version, which Nvidia calls SIMT (single instruction, multiple threads), goes further by letting each lane behave as if it were an independent program with its own variables and its own place in the code, while underneath, the hardware runs them in groups of 32 in lockstep. A group of 32 is called a warp, the term has been in every Nvidia GPU since 2006, and it is the single most useful concept for understanding anything about how a GPU behaves. Part 7 is built around it.
Where this leaves us
The GPU is not a faster CPU. It is a different bet about the shape of work: that the job is wide, that the items are independent, and that nobody minds if any one of them takes a while. Graphics has that shape, which is why the bet was made in the first place. It then turned out, more or less by luck, that neural networks have exactly the same shape, and a chip designed to colour pixels found itself the best available machine for w × x + b at scale. That is the whole reason a graphics company is at the centre of AI.
The bet has a cost. A GPU cannot run your operating system, your browser, or the sequential parts of any program, and it never will. Every GPU system has a CPU beside it to do the chain and to tell the GPU what to do, a relationship Part 10 covers. Before that, the next three parts open up the GPU itself: the unit that holds the 32 calculators, the memory that feeds them, and the specialised unit that does matrix arithmetic faster than the lanes ever could.
Next: Inside a Streaming Multiprocessor: the actual building block of a GPU, the warp scheduler that makes latency tolerance work, and what happens when 32 threads in lockstep disagree about an if.