The toy processor in Part 5 told a small lie. When it executed LOAD R1, [w], the number appeared in the register on the next tick, as if memory were as close as the adder. On a real machine that instruction is the slowest thing in the program by a mile. A multiply takes a handful of ticks. A load from main memory takes somewhere around a hundred nanoseconds, which at a few gigahertz is several hundred ticks. The processor could have done a few hundred multiply-adds in the time it spent waiting for one number to arrive.
This gap has a name, the memory wall, and it is the single most important fact about processor design of the last thirty years. Almost everything strange about a CPU, and almost everything strange about a GPU, is a response to it. Part 6 set out how the two machines differ in shape. This part is about why they took those shapes, and if you understand it, the rest of the series is mostly consequences.
Why the filing cabinet is far away
The registers of Part 5 are built from six-transistor SRAM cells sitting right beside the arithmetic. Main memory is not. It is DRAM, one transistor and one capacitor per bit, on separate chips, connected to the processor by wires that are centimetres long. Reaching it means sending an address out over those wires, waiting for the DRAM chip to select a whole row of cells, waiting for a set of very sensitive amplifiers to work out whether each tiny capacitor holds a charge or not, and then sending the data back. Each step takes tens of nanoseconds, and none of them has got much faster in twenty years, because they are limited by physics rather than by transistor count. A capacitor small enough to fit billions to a chip holds so little charge that reading it simply takes time.
This is where the distinction between latency and bandwidth becomes essential, and it will come up in every remaining part. Latency is how long one request takes from asking to receiving. Bandwidth is how much data arrives per second once things are flowing. They are independent. The classic illustration is a lorry full of hard drives driving across the country: the bandwidth is astonishing, petabytes per day, and the latency is a day. Memory systems are like that. You can raise bandwidth almost at will, by making the bus wider, adding more channels, or stacking more chips, and every generation of memory does. You cannot do much about latency. DRAM latency has hovered around the same tens of nanoseconds since the late 1990s while processors got dozens of times faster.
The ladder
The standard answer is to keep copies of the most-used data somewhere closer. A cache is a block of SRAM, near the processor, that holds a copy of recently used pieces of main memory. When the processor asks for an address, the cache checks whether it has a copy. If it does (a hit), the data comes back in a few ticks. If not (a miss), the request goes on to the next level, and the data is copied into the cache on its way back so that the next access to it is fast.
Caches come in layers, each bigger and slower than the last, and the whole arrangement is called the memory hierarchy. The analogy that fits best is where you keep documents. A few numbers on your desk, the registers. A shelf behind your chair, the first-level cache. A cabinet in the corridor, the second level. The department library, the third. The city archive across town, main memory. And a warehouse in another country, the disk.
| Level | Built from | Typical size | Roughly how far, in ticks |
|---|---|---|---|
| Registers | SRAM | a few hundred bytes per thread | 1 |
| L1 cache | SRAM | tens of kilobytes | a few |
| L2 cache | SRAM | megabytes | ten to twenty on a CPU, a couple of hundred on a GPU |
| L3 cache | SRAM | tens of megabytes (CPU only) | forty or so |
| Main memory | DRAM | gigabytes | hundreds |
| SSD | flash | terabytes | hundreds of thousands |
Side note: which of these still use the capacitor? Every kind of memory in the table between the caches and the SSD is DRAM, and it is the same one-transistor-one-capacitor cell from Part 5 in every case, only packaged differently. The sticks of DDR5 in a desktop, the GDDR7 chips soldered around a GPU, and the HBM stacks bonded next to an H100 or Blackwell die all store each bit as a charge that leaks and has to be refreshed every 64 milliseconds, and all of them forget everything the instant the power goes. What they differ in is how they are wired to the processor: how many wires, how fast each wire signals, and how close the chips sit, which Part 6 showed and Part 10 comes back to. An SSD is a different device altogether. It is built from NAND flash, and the name is the NAND gate from Part 4, because the cells in a flash string are wired in series like the transistors of that gate. A flash cell is a transistor with a second, buried layer under its control gate, insulated on every side, either a small floating gate or a thin film that traps charge. Writing forces electrons through the insulator onto that layer, where they stay put for years with no power at all, and their presence or absence shifts the voltage at which the transistor turns on, which is how the bit is read back. No capacitor, no refresh, nothing lost when the power goes, and tens of times denser and cheaper per bit than DRAM. The price is time and wear: a read takes tens of microseconds rather than tens of nanoseconds, a write takes hundreds, and each cell survives only a few thousand rewrites before its insulator gives out. That is why flash is the warehouse and DRAM is the archive across town, and why nothing a GPU does in the middle of a calculation touches an SSD if the programmer can help it.
Caches work because of a property of real programs called locality. If you used a number just now, you will probably use it again soon (temporal locality), and if you used one number you will probably use its neighbours next (spatial locality). So a cache does not fetch one byte at a time. It fetches a line, a chunk of 32 to 128 adjacent bytes, on the assumption that the neighbours will be wanted. Most of the time this is right, and a well-behaved program finds its data in cache more than ninety-five per cent of the time. The remaining few per cent, the misses, dominate its running time anyway, because each one costs hundreds of ticks.
The phrase "memory wall" comes from a short 1995 paper by Wulf and McKee that pointed out the arithmetic: processor speed was improving by around fifty per cent a year and memory latency by around seven, so however good the caches got, the time spent on misses would eventually be all the time there was. They were right, and the two answers the industry found are the subject of the rest of this part.
What the wait costs on a GPU
The numbers below come from a group of researchers who measured a Volta V100 with microbenchmarks, published as Dissecting the NVIDIA Volta GPU Architecture. Later chips differ in detail but the shape is the same. Click a level to see how many multiply-adds one lane could have done in the time it waited.
Measured latency of one access at each level of a V100's memory hierarchy, in clock ticks, and what that wait is worth in arithmetic.
Click a row.
Two things stand out. The steps are not evenly spaced. Each rung of the ladder is several times slower than the last, and the jump from L2 to main memory is the cliff edge. And the GPU's numbers are, if anything, worse than a CPU's. A CPU reaches main memory in roughly 300 ticks. The GPU takes 375 in the best case. GPU memory is not built for low latency. It is built for bandwidth, and that choice only makes sense given how the GPU responds to the wall.
The CPU's answer: never let one thread wait
A CPU is designed to make a single stream of instructions run as fast as possible, so its strategy is to hide latency. It has large caches, tens of megabytes of L3, to make misses rare. It has prefetchers that watch the pattern of addresses and fetch the next lines before they are asked for. And when a miss does happen, it does not simply stop. Here is what a modern core actually does in the hundreds of ticks while a number is on its way. The load is marked as outstanding and the core carries on fetching and decoding, up to a few hundred instructions ahead, into the reorder buffer from Part 5. Any of those that do not need the missing number, arithmetic on other registers, address calculations, other loads, get executed straight away, out of order, and their results are parked until they can be filed in the right sequence. Other loads found this way are sent off too, so a core can have ten or twenty trips to memory in flight at once rather than one after another. The branch predictor keeps supplying the likely path so the fetching never pauses. And if the core is running a second thread on the same hardware, which is what "hyper-threading" means, that thread's instructions fill any gaps the first one leaves. All of this works until the reorder buffer is full of instructions that are all waiting on the one missing number, and then the core stalls with nothing left to do. Every piece of this is expensive in transistors, and every piece is there so that when the one thread hits a miss, useful work still gets done for as long as possible.
This is a superb strategy for the kind of program a CPU usually runs: a word processor, a web browser, a compiler, a game's logic. Those programs are long chains of dependent decisions, where the next instruction depends on the result of the last, and there is no other work to switch to. The CPU is a specialist in doing one thing at a time without stalling.
The GPU's answer: always have someone else ready
A GPU makes the opposite bet. It does not try to hide latency for any one thread. It tolerates latency by keeping thousands of threads in flight, so that when one of them issues a load and has to wait 400 ticks, the scheduler simply switches to another thread that has its data and is ready to go. Concretely: each warp scheduler from Part 6, the front end that feeds 32 lanes, has a dozen or more warps assigned to it, every one with its own registers already loaded in the big register file, and at every tick it picks any warp whose next instruction is ready to run. A warp that has just asked for a number from memory is marked not ready and simply skipped, at no cost, because nothing has to be saved or restored. By the time the scheduler gets back round to it, the data has arrived. No thread is fast. The machine as a whole never idles, as long as there are enough warps to go round.
What happens during the wait. Three machines each run a program that does a few ticks of arithmetic, then asks memory for the next number, then waits. Green is arithmetic getting done, pale is a calculator standing idle. Move the sliders and watch how much of the time each machine is actually working.
You can see this bet in the transistor budget. A GPU's caches are small for its size, because caches are for hiding latency and the GPU has another way. What it has instead is an enormous register file: each streaming multiprocessor on an Ada chip holds 256 KB of registers, twice the size of its L1 cache, because every one of those thousands of resident threads needs its own numbers kept on the desk, ready to resume the instant it is scheduled. And its memory system is built for bandwidth rather than latency: a very wide bus to GDDR, or stacked HBM sitting on the same package, delivering a terabyte or more per second, because with thousands of threads all asking for data at once, the question is not how fast one request returns but how many requests per second the system can satisfy.
The strategy has a condition attached. It only works if there are thousands of independent threads to switch between. Colouring pixels is like that. So is multiplying matrices, and so, it turns out, is nearly everything in a neural network. A long chain of dependent decisions is not, and on that kind of work a GPU is helpless: every thread waits and there is nobody to switch to.
Where this leaves us
Both designs are answers to the same wall. The CPU spends its transistors on making one thread never wait, and the GPU spends them on having so many threads that waiting does not matter. Neither is smarter. They are optimised for different shapes of work, and Part 8 puts them side by side on the same task to make the difference visible.
Keep one number from this part above all others: hundreds of ticks per trip to memory, against four per multiply-add. Everything that follows in this series, the warp schedulers, the shared memory, the tensor cores, HBM, NVLink, all of it, is machinery for feeding an arithmetic unit that would otherwise sit idle ninety-nine per cent of the time.
Next: Two Ways to Be Fast: a race between a CPU and a GPU on the same job, where each wins, and why the shape of a neural network is the GPU's home ground.