The toy processor in Part 4 told a small lie. When it executed LOAD R1, [w], the number appeared in the register on the next tick, as if memory were as close as the adder. On a real machine that instruction is the slowest thing in the program by a mile. A multiply takes a handful of ticks. A load from main memory takes somewhere around a hundred nanoseconds, which at a few gigahertz is several hundred ticks. The processor could have done a few hundred multiply-adds in the time it spent waiting for one number to arrive.
This gap has a name, the memory wall, and it is the single most important fact about processor design of the last thirty years. Almost everything strange about a CPU, and almost everything strange about a GPU, is a response to it. If you understand this part, the rest of the series is mostly consequences.
Why the filing cabinet is far away
The registers of Part 4 are built from six-transistor SRAM cells sitting right beside the arithmetic. Main memory is not. It is DRAM, one transistor and one capacitor per bit, on separate chips, connected to the processor by wires that are centimetres long. Reaching it means sending an address out over those wires, waiting for the DRAM chip to select a whole row of cells, waiting for a set of very sensitive amplifiers to work out whether each tiny capacitor holds a charge or not, and then sending the data back. Each step takes tens of nanoseconds, and none of them has got much faster in twenty years, because they are limited by physics rather than by transistor count. A capacitor small enough to fit billions to a chip holds so little charge that reading it simply takes time.
This is where the distinction between latency and bandwidth becomes essential, and it will come up in every remaining part. Latency is how long one request takes from asking to receiving. Bandwidth is how much data arrives per second once things are flowing. They are independent. The classic illustration is a lorry full of hard drives driving across the country: the bandwidth is astonishing, petabytes per day, and the latency is a day. Memory systems are like that. You can raise bandwidth almost at will, by making the bus wider, adding more channels, or stacking more chips, and every generation of memory does. You cannot do much about latency. DRAM latency has hovered around the same tens of nanoseconds since the late 1990s while processors got dozens of times faster.
The ladder
The standard answer is to keep copies of the most-used data somewhere closer. A cache is a block of SRAM, near the processor, that holds a copy of recently used pieces of main memory. When the processor asks for an address, the cache checks whether it has a copy. If it does (a hit), the data comes back in a few ticks. If not (a miss), the request goes on to the next level, and the data is copied into the cache on its way back so that the next access to it is fast.
Caches come in layers, each bigger and slower than the last, and the whole arrangement is called the memory hierarchy. The analogy that fits best is where you keep documents. A few numbers on your desk, the registers. A shelf behind your chair, the first-level cache. A cabinet in the corridor, the second level. The department library, the third. The city archive across town, main memory. And a warehouse in another country, the disk.
| Level | Built from | Typical size | Roughly how far, in ticks |
|---|---|---|---|
| Registers | SRAM | a few hundred bytes per thread | 1 |
| L1 cache | SRAM | tens of kilobytes | a few |
| L2 cache | SRAM | megabytes | ten to twenty on a CPU, a couple of hundred on a GPU |
| L3 cache | SRAM | tens of megabytes (CPU only) | forty or so |
| Main memory | DRAM | gigabytes | hundreds |
| SSD | flash | terabytes | hundreds of thousands |
Caches work because of a property of real programs called locality. If you used a number just now, you will probably use it again soon (temporal locality), and if you used one number you will probably use its neighbours next (spatial locality). So a cache does not fetch one byte at a time. It fetches a line, a chunk of 32 to 128 adjacent bytes, on the assumption that the neighbours will be wanted. Most of the time this is right, and a well-behaved program finds its data in cache more than ninety-five per cent of the time. The remaining few per cent, the misses, dominate its running time anyway, because each one costs hundreds of ticks.
The phrase "memory wall" comes from a short 1995 paper by Wulf and McKee that pointed out the arithmetic: processor speed was improving by around fifty per cent a year and memory latency by around seven, so however good the caches got, the time spent on misses would eventually be all the time there was. They were right, and the two answers the industry found are the subject of the rest of this part.
What the wait costs on a GPU
The numbers below come from a group of researchers who measured a Volta V100 with microbenchmarks, published as Dissecting the NVIDIA Volta GPU Architecture. Later chips differ in detail but the shape is the same. Click a level to see how many multiply-adds one lane could have done in the time it waited.
Measured latency of one access at each level of a V100's memory hierarchy, in clock ticks, and what that wait is worth in arithmetic.
Click a row.
Two things stand out. The steps are not evenly spaced. Each rung of the ladder is several times slower than the last, and the jump from L2 to main memory is the cliff edge. And the GPU's numbers are, if anything, worse than a CPU's. A CPU reaches main memory in roughly 300 ticks. The GPU takes 375 in the best case. GPU memory is not built for low latency. It is built for bandwidth, and that choice only makes sense given how the GPU responds to the wall.
The CPU's answer: never let one thread wait
A CPU is designed to make a single stream of instructions run as fast as possible, so its strategy is to hide latency. It has large caches, tens of megabytes of L3, to make misses rare. It has prefetchers that watch the pattern of addresses and fetch the next lines before they are asked for. It has the out-of-order machinery from Part 4, which lets it keep executing instructions that do not depend on the outstanding load, and branch prediction so it knows which instructions those will be. Every one of these is expensive in transistors, and every one is there so that when the one thread hits a miss, useful work still gets done.
This is a superb strategy for the kind of program a CPU usually runs: a word processor, a web browser, a compiler, a game's logic. Those programs are long chains of dependent decisions, where the next instruction depends on the result of the last, and there is no other work to switch to. The CPU is a specialist in doing one thing at a time without stalling.
The GPU's answer: always have someone else ready
A GPU makes the opposite bet. It does not try to hide latency for any one thread. It tolerates latency by keeping thousands of threads in flight, so that when one of them issues a load and has to wait 400 ticks, the scheduler simply switches to another thread that has its data and is ready to go. By the time it gets back round to the first thread, the data has arrived. No thread is fast. The machine as a whole never idles.
You can see this bet in the transistor budget. A GPU's caches are small for its size, because caches are for hiding latency and the GPU has another way. What it has instead is an enormous register file: each streaming multiprocessor on an Ada chip holds 256 KB of registers, twice the size of its L1 cache, because every one of those thousands of resident threads needs its own numbers kept on the desk, ready to resume the instant it is scheduled. And its memory system is built for bandwidth rather than latency: a very wide bus to GDDR, or stacked HBM sitting on the same package, delivering a terabyte or more per second, because with thousands of threads all asking for data at once, the question is not how fast one request returns but how many requests per second the system can satisfy.
The strategy has a condition attached. It only works if there are thousands of independent threads to switch between. Colouring pixels is like that. So is multiplying matrices, and so, it turns out, is nearly everything in a neural network. A long chain of dependent decisions is not, and on that kind of work a GPU is helpless: every thread waits and there is nobody to switch to.
Where this leaves us
Both designs are answers to the same wall. The CPU spends its transistors on making one thread never wait, and the GPU spends them on having so many threads that waiting does not matter. Neither is smarter. They are optimised for different shapes of work, and Part 6 puts them side by side on the same task to make the difference visible.
Keep one number from this part above all others: hundreds of ticks per trip to memory, against four per multiply-add. Everything that follows in this series, the warp schedulers, the shared memory, the tensor cores, HBM, NVLink, all of it, is machinery for feeding an arithmetic unit that would otherwise sit idle ninety-nine per cent of the time.
Next: Two Ways to Be Fast: a race between a CPU and a GPU on the same job, where each wins, and why the shape of a neural network is the GPU's home ground.