Inside a Streaming Multiprocessor

Look at a photograph of the AD102 die, the chip inside an RTX 4090, and the first impression is of a city seen from the air. There is a grid of large identical blocks, twelve of them, each subdivided into a dozen smaller identical blocks, with a broad band of something else running down the middle. The big blocks are graphics processing clusters (GPCs). The small blocks inside them are streaming multiprocessors (SMs), and there are 144 on the full die. The band down the middle is the L2 cache, which is Part 8's business.

The SM is the unit that matters. When Nvidia says a GPU has 16,384 CUDA cores, what it means is 128 SMs with 128 lanes each. When it says a new generation is faster, most of the change is inside the SM. And when you want to understand why a GPU does well on one job and badly on another, the SM is where the answer lives. This part opens one up.

The floor plan

An SM on Ada or Hopper is split into four identical quarters, which Nvidia calls processing blocks or partitions. Each quarter has its own warp scheduler, its own bank of 32 lanes for ordinary floating-point and integer arithmetic (Ada makes 16 of them capable of integer work as well), its own quarter of the register file, one tensor core, and a few units for loading and storing data. Shared among the four quarters are a block of fast memory that serves as both L1 cache and programmer-controlled scratchpad (128 KB on Ada, 256 KB on Hopper), the special function units that compute things like sine, exponential, and reciprocal, and on a graphics chip an RT core for ray tracing.

Add up the four quarters and one SM has 128 lanes, four tensor cores, and 256 KB of registers. Multiply by 128 SMs and you have the RTX 4090's 16,384 CUDA cores and 32 MB of register file, which is more fast on-chip storage than most CPUs have cache. That was Part 5's point made concrete: the GPU spends its silicon on keeping thousands of threads' numbers on the desk, not on caches.

Warps, and the scheduler's tick

Now the part that makes it all work. A warp is a group of 32 threads that the hardware handles as one unit. Each thread in the warp is, from the programmer's point of view, its own little program with its own variables. From the hardware's point of view, a warp is one instruction stream, and when the scheduler issues an instruction, all 32 lanes carry it out at once, each on its own thread's registers. This is the one-clerk-and-32-calculators arrangement from Part 6, and the number 32 has been baked into every Nvidia GPU since the G80 in 2006.

Each partition can hold many warps at the same time, not just one. On Ada an SM can keep 48 warps resident, twelve per partition, which is 1,536 threads. Hopper stretches that to 64 warps and 2,048 threads. Every resident warp has its registers permanently allocated in the register file, which is why the register file is so large. And on every tick, each scheduler does the same simple thing: look at its dozen resident warps, find one whose next instruction is ready to run, and issue it.

That single sentence is the GPU's answer to the memory wall. Suppose warp 3 executes a load from memory, and the data will take 400 ticks to arrive. The scheduler does not wait. On the next tick it looks again, finds that warp 7 has a multiply ready, and issues that. Then warp 1, then warp 9, round and round. Warp 3's registers sit untouched in the file, so when its data finally arrives the scheduler can pick it straight back up. There is no context switch, no saving and restoring, no cost at all. As long as there is always some warp ready, the lanes never idle, and the 400-tick wait is invisible.

The fraction of an SM's warp slots that are actually filled is called occupancy, and it is the first thing a GPU programmer looks at when something runs slowly. Low occupancy means few warps to switch between, which means memory stalls start showing through. The usual cause is greed: if each thread uses a lot of registers, fewer threads fit in the file. At full occupancy on Ada, 65,536 registers shared among 1,536 threads leaves each thread about 42. A thread that wants 100 registers cuts the SM's occupancy by more than half.

Here is the classroom picture I use, since it holds up well. The SM is a classroom with four teachers. A warp is a row of 32 pupils. Each teacher reads out one instruction, "multiply the number on your sheet by the one next to it", and every pupil in the row does it to their own sheet. When a row has sent to the library for a book and is waiting, the teacher simply turns to another row. The place the analogy breaks is the important one: real pupils could get on with something else while they wait, whereas a lane cannot. Only the scheduler can switch, and only between whole rows.

When the row disagrees

Lockstep has a cost, and it shows up the moment the 32 threads of a warp reach an if and do not all agree. Suppose the instruction is "if your number is positive, do step A, otherwise do step B", and twenty of the threads have positive numbers. The hardware cannot send twenty pupils one way and twelve the other. There is one instruction stream. So it runs step A with the twelve non-positive lanes switched off, and then runs step B with the twenty positive lanes switched off. Both paths execute, one after the other, and the warp takes as long as A plus B. This is branch divergence, and it is the price of sharing one clerk across 32 calculators.

Notice that the split does not matter. One dissenting thread costs as much as sixteen, because the whole of B has to run either way. Efficient GPU code is written so that the threads of a warp make the same decisions, or so that the decisions are made by whole warps at once. Graphics has always been like this, because neighbouring pixels usually do the same thing, and matrix arithmetic has no branches at all, which is one more reason a neural network suits the machine. Since Volta the hardware has been able to interleave the two paths rather than strictly finishing one before the other, which fixes some awkward cases, but the cost in lane-time is unchanged.

From SM to die

The rest of the die is organisation. Two SMs make a texture processing cluster (TPC), six TPCs make a GPC, twelve GPCs make an AD102. When a program launches work on the GPU, a unit called the GigaThread engine hands out blocks of threads to SMs that have room, and refills them as they finish. A GPU product is rarely the full die: the RTX 4090 has 128 of the 144 SMs enabled and the H100 has 132 of 144, because some SMs on each manufactured chip have defects, and disabling a few lets more chips be sold rather than thrown away.

Where this leaves us

A GPU is a few dozen to a couple of hundred copies of one building block, and that block is the SM: four schedulers, each issuing one instruction per tick to a warp of 32 lanes, choosing among a dozen resident warps so that the arithmetic never has to wait for memory. Everything Nvidia has done with the SM for a decade is variation on this theme, and the counts on a spec sheet, CUDA cores, tensor cores, threads per SM, register file size, are all just multiples of it.

What the SM cannot do on its own is keep itself fed. Each lane does a multiply-add every tick, and every one of those needs two numbers in and one out. Across a whole chip that is tens of trillions of numbers a second, and the memory of Part 5 delivers nowhere near that. How the gap gets closed is the next part, and it is where the design of a GPU becomes a design of its memory.

Next: Feeding the Cores: the GPU memory hierarchy from registers to HBM, the roofline that tells you whether your job is starved of arithmetic or of bytes, and why a language model generating text is almost always the second.