Twenty-two parts ago I opened a microwave oven, found a vacuum tube inside, and asked how we got from that to a machine with 76 billion switches that draws games and runs language models. The answer took a while, because it is a stack of ideas rather than a single trick, and each layer only makes sense once the one below it is in place. This part puts the stack back together in one picture, then hands you the skill the series was really for: picking up any GPU spec sheet and knowing what every line means and which ones matter for your job. It ends with a glossary. This is the part to bookmark.
The stack, bottom to top
The stack above is the chain of containment that the running calculation passes through. Here is the same stack drawn the other way, as block diagrams, so that the parts that are not on the multiply-add's direct path, the caches, the switches, the schedulers, the tensor and RT cores, are in the picture too. Each panel opens up the highlighted block of the one above it.
Read it upward and it is the series in order. Below the bottom box sit the relay and the vacuum tube, which did the transistor's job first and could not be made small, cool, or numerous enough to build this stack from. A transistor is a switch. A dozen switches make a gate. Thousands of gates make a unit that computes w × x + b. Wrap that in registers and you have a lane. Put 128 lanes under four schedulers, each reading one instruction to 32 lanes at once and switching between warps whenever one waits for memory, and you have an SM. Tile 144 SMs around a cache and you have a die. Join two dies and surround them with stacked memory and you have a package. Wire 72 packages together at 1.8 TB/s each and you have the machine that trains a language model.
Read it downward and it is the memory wall. At every level the arithmetic is cheap and getting the numbers to it is the problem, and every level's design is a way of moving fewer bytes: registers instead of memory, shared memory tiles, the L2, HBM on the package, NVLink instead of PCIe, fewer bits per number in the tensor cores. If you remember one sentence from the twenty-three parts, make it the one from Part 10: compute is cheap, bytes are expensive.
How to read a spec sheet
Every line on a GPU spec sheet corresponds to something in this series. Here is the H100 SXM's, line by line, with what each one is really telling you.
| Line on the sheet | What it means | Where |
|---|---|---|
| TSMC 4N, 80 billion transistors | the process generation (a marketing name) and the switch count, which sets a hard ceiling on everything else | Part 3, 15 |
| 814 mm² die, 132 of 144 SMs enabled | the printed area, which fixes how many fit on a wafer and how many are clean, and the yield story made visible: the SMs that failed were fused off | Part 3, 15 |
| 132 SMs, 16,896 CUDA cores | 132 copies of the building block, each with 128 lanes. "CUDA cores" is lanes, not cores in the CPU sense | Part 9 |
| Boost clock 1.98 GHz | ticks per second. Multiply by lanes and by two for the FP32 peak | Part 5 |
| 528 tensor cores, 4th generation | four per SM. The generation tells you which precisions they support | Part 11 |
| FP32 67 TFLOPS | the plain lanes' peak. Rarely the number that matters for AI | Part 9 |
| FP16 989 TFLOPS, FP8 1,979 TFLOPS, "with sparsity" 2× | tensor core peaks. Always find the dense figure and compare like with like | Part 11 |
| 80 GB HBM3, 3.35 TB/s | how much model fits, and how many times a second it can be read. For generation, this is the line that matters most | Part 10 |
| 50 MB L2 | the chip-wide cache that catches reuse before it goes off-die | Part 10 |
| 228 KB shared memory per SM | the programmer-managed tile buffer. Bigger means bigger tiles for the tensor cores | Part 10 |
| NVLink 900 GB/s, PCIe Gen 5 128 GB/s | the roads to other GPUs and to the CPU. About seven to one between them | Part 12, 13 |
| Compute capability 9.0 | which CUDA features the chip has. 9.0 means clusters, TMA, FP8 | Part 12 |
| 700 W | the power budget for the SXM module. Also the cooling problem, and the datacentre's electricity bill | Part 21 |
The same lines with different numbers describe any GPU from the last decade. A gaming card adds RT cores, ROPs, texture units, video encoders, and display outputs, the pipeline hardware of Part 16 that Part 17 covered. A newer data centre part adds dies per package and a rack-level line for the NVLink domain.
Three questions for any chip
When someone asks whether a given GPU is a good fit for a given job, the spec sheet reduces to three questions, and they are the roofline of Part 10 in words.
How many multiply-adds per second can it do, at the precision the job will actually use, without sparsity? That is the flat roof.
How many bytes per second can it move, from its own memory for one GPU, and from its neighbours over NVLink if the model is split? That is the sloping roof.
How many operations does the job do per byte it moves? A chatbot generating text does about one per byte at 16-bit precision. A training run or a long-prompt prefill does thousands. A single-user workload sits far left and is limited entirely by the second question, and the fix is fewer bytes: quantise, batch, or buy bandwidth. A batched or training workload sits right and is limited by the first, and the fix is more tensor cores or fewer bits per number.
Everything else on the sheet is detail that adjusts one of those three numbers. That is the whole method, and it is the one I now use before I read any benchmark.
Five chips on one page
| RTX 4090 (Ada, 2022) | RTX 5090 (Blackwell, 2025) | H100 SXM (Hopper, 2022) | B200 (Blackwell, 2024) | Rubin (2026, stated) | |
|---|---|---|---|---|---|
| Transistors | 76.3 billion | 92.2 billion | 80 billion | 208 billion, 2 dies | 336 billion, 2 dies |
| SMs | 128 | 170 | 132 | not published | 224 |
| Memory | 24 GB GDDR6X | 32 GB GDDR7 | 80 GB HBM3 | up to 192 GB HBM3e | 288 GB HBM4 |
| Bandwidth | 1.0 TB/s | 1.79 TB/s | 3.35 TB/s | 8 TB/s | 22 TB/s target |
| FP8 tensor, dense | 660 TFLOPS | about 830 TFLOPS | 1,979 TFLOPS | 4.5 PFLOPS | 17.5 PFLOPS |
| FP4 tensor | no | about 1.7 PFLOPS dense | no | 9 PFLOPS dense | 50 PFLOPS, stated |
| GPU-to-GPU link | PCIe only | PCIe only | 900 GB/s | 1.8 TB/s | 3.6 TB/s |
| Power | 450 W | 575 W | 700 W | 1,000 to 1,200 W | not published |
The RTX 5090's tensor figures are derived from Nvidia's "AI TOPS" number, which is an FP4 figure quoted with sparsity (Part 22 explains TOPS), and should be read as approximate. The Rubin column is Nvidia's own preliminary figures, as Part 20 explained. Look at the bandwidth row and then the FP4 row, and you are looking at what the last four years of GPU design were about.
Glossary
| Term | Meaning |
|---|---|
| Relay | an electromagnet that moves a contact. The oldest electrically controlled switch, slow and big, still used wherever big currents need switching with total isolation. (Part 1) |
| Vacuum tube | a heated cathode, a grid, and a plate in a vacuum. The first amplifier and the switch of the first computers, limited by heat, size, and burnout. Still used for extreme power and frequency. (Part 2) |
| Tyranny of numbers | the 1950s wall where every extra tube or soldered joint added cost, heat, and failures. Solved by the transistor and then the integrated circuit. (Part 2) |
| Transistor | a switch flipped by voltage on its gate. The only component in a chip. (Part 3) |
| Wafer | the 300 mm disc of pure silicon on which chips are printed, the unit the whole industry counts in. (Part 3, 15) |
| Die | one printed copy of a chip on the wafer, cut out and packaged. The thing whose area and transistor count a spec sheet quotes. (Part 3, 15) |
| Yield, binning | the share of dies that come out clean, and the sorting of the rest into products with units fused off. Why the 4090 has 128 of 144 SMs. (Part 15) |
| Reticle limit | the largest die a lithography machine can print in one exposure, 858 mm². Why Blackwell has two dies. (Part 3, 15, 19) |
| Package, interposer, CoWoS | the carrier that connects a die to the board, the slab of silicon wiring that lets HBM sit beside the GPU, and TSMC's name for the combination. (Part 15) |
| HBM stack, TSV | eight or twelve memory dies stacked and joined by holes through the silicon, so a fingernail of DRAM offers a thousand wires. (Part 10, 15) |
| Frame, frame buffer | one full screen of pixel colours, and the memory that holds it. (Part 16) |
| Vertex, triangle, shader | a corner with a position and a few attributes, the only shape a GPU draws, and a small program run once per vertex or per pixel. (Part 16) |
| Rasteriser, ROP, texture unit | the fixed-function units that find the pixels a triangle covers, write finished pixels, and sample pictures onto surfaces. (Part 16) |
| Dennard scaling | the rule, broken around 2005, that smaller transistors ran faster for the same power. Its end is why chips went wide instead of fast. (Part 3) |
| Logic gate | a few transistors implementing a truth table: AND, OR, XOR, NOT, NAND. (Part 4) |
| FMA | fused multiply-add, a × b + c in one operation with one rounding. The atom of GPU arithmetic. (Part 4) |
| FP32, FP16, BF16, FP8, FP4 | floating-point formats named by total bits. Fewer bits means a smaller multiplier and fewer bytes to move. (Part 4, 11) |
| Register | a small bank of SRAM next to the arithmetic, reachable in one tick. (Part 5) |
| Clock | the signal that ticks every register at once. Its rate is capped by the slowest chain of gates. (Part 5) |
| Latency, bandwidth | how long one access takes, and how much data arrives per second. Independent, and only the second is easy to improve. (Part 7) |
| Cache, L1, L2 | copies of recently used memory kept close by. GPUs have small L1s and, recently, large L2s. (Part 7, 10) |
| Memory wall | the growing gap between how fast a processor can compute and how fast memory can feed it. (Part 7) |
| Latency-oriented, throughput-oriented | the CPU's and GPU's opposite strategies for the memory wall: never let one thread wait, or always have another thread ready. (Part 7, 8) |
| Embarrassingly parallel | work made of independent items, like pixels or a layer of neurons. What GPUs are for. (Part 8) |
| Amdahl's law | the sequential fraction of a job caps its speedup no matter how many lanes you add. (Part 8) |
| SIMT | single instruction, multiple threads. One instruction issued to 32 lanes, each with its own data. (Part 8, 9) |
| Warp | 32 threads executed in lockstep. The unit the scheduler works in, unchanged since 2006. (Part 9) |
| SM | streaming multiprocessor. Four schedulers, 128 lanes, tensor cores, shared memory. The building block. (Part 9) |
| CUDA core | Nvidia's name for one lane. Not a core in the CPU sense. (Part 8, 9) |
| Occupancy | how many warps an SM has resident to switch between. Low occupancy means memory stalls show through. (Part 9) |
| Branch divergence | threads of a warp disagreeing at an if, forcing both paths to run with lanes masked off. (Part 9) |
| Shared memory | per-SM SRAM the program manages directly, used to hold tiles for reuse. (Part 10) |
| GDDR, HBM | graphics memory on the board, or stacked memory on the package. Cheap and wire-limited, or expensive and very wide. (Part 10) |
| Arithmetic intensity | operations per byte moved. The one number that says whether a job is memory-bound or compute-bound. (Part 10) |
| Roofline | the diagram of peak compute (flat) and bandwidth (sloped) that shows a job's ceiling at its intensity. (Part 10) |
| Coalescing | a warp's 32 adjacent loads served by one memory transaction. Scattered loads cost up to 32. (Part 10) |
| Tensor core | a unit that multiplies small matrix tiles in one instruction, with reuse built into its wiring. (Part 11) |
| Sparsity, 2:4 | skipping zeros when two of every four weights are zero. Doubles the headline number, rarely applies. (Part 11) |
| Transformer Engine | Hopper's hardware and software for choosing scale factors so that FP8, and on Blackwell FP4, can be used safely. (Part 11, 18, 19) |
| Host, device | the CPU and its memory, the GPU and its memory. (Part 12) |
| Kernel, grid, block | a function written for one thread, launched across a grid of blocks, each block living on one SM. (Part 12) |
| PCIe | the slot that connects host and device, 32 to 64 GB/s. The slow road. (Part 12) |
| PTX, compute capability | the portable intermediate code CUDA compiles to, and the version number of a GPU's feature set. (Part 12) |
| NVLink, NVSwitch | the dedicated GPU-to-GPU link, and the switch that lets every GPU in a node or rack reach every other. (Part 13) |
| NVLink domain | the set of GPUs joined by NVSwitch, currently 72, that software can treat as one memory. (Part 13, 19) |
| Data, tensor, pipeline parallelism | the three ways to cut a model across GPUs, from least to most communication. (Part 13) |
| Superchip, NVLink-C2C | a CPU and GPU on one board joined by a coherent link, so the GPU can spill into CPU memory. (Part 13) |
| Thread block cluster, TMA | Hopper's additions: blocks that share each other's shared memory, and a unit that moves tiles without lane instructions. (Part 18) |
| NV-HBI | the 10 TB/s seam joining Blackwell's two dies into one GPU. (Part 19) |
| Prefill, decode | the compute-bound prompt-reading phase and the memory-bound generating phase of an LLM request. (Part 10, 20) |
| Disaggregated inference | running prefill and decode on different machines, each with the memory it needs. (Part 20) |
| TDP, TGP, SXM | the power a chip is designed to shed continuously, Nvidia's name for it on gaming cards, and the module form factor that allows 700 W where a PCIe card allows 350. (Part 21) |
| Dynamic and static power | the energy of flipping switches, and the leakage through switches that are off. Both end up as heat. (Part 21) |
| PUE | power usage effectiveness: the whole building's draw divided by the computers' draw. Industry average about 1.56, the best about 1.1. (Part 21) |
| WUE, withdrawal, consumption | water usage effectiveness: litres of water consumed on site per kWh the computers use. Water that evaporates in a cooling tower is consumed, water returned to the river is only withdrawn. (Part 21) |
| Direct-to-chip liquid cooling | coolant piped through a cold plate on each chip, needed once a rack passes a few tens of kilowatts. (Part 21) |
| SoC, unified memory | a phone or laptop's CPU, GPU and NPU on one die sharing one pool of memory. (Part 22) |
| NPU | neural processing unit: a fixed-function matrix engine on a system-on-chip, for running trained networks at a few watts. (Part 22) |
| Systolic array | a grid of multiply-add cells through which inputs and partial sums flow, each number fetched once. The heart of an NPU and of the TPU. (Part 22) |
| TOPS | trillions of operations per second, usually at 8-bit integer precision. Always ask how many bits, dense or sparse, and at what power. (Part 22) |
| TPU | Google's tensor processing unit: a systolic array grown to data-centre scale, with HBM and an interconnect around it. (Part 22) |
What I took from the trip
Jon Stokes's book gave me the calculator and the file clerk, and for years that was my whole model of a processor. What this series added, for me, is one more figure in the room: the clerk who reads each instruction aloud to 32 calculators at once, and keeps a dozen rows of them on the go so that whenever one row is waiting for the library, another is working. Everything distinctive about a GPU is that figure. The warp, the register file the size of a CPU's cache, the small caches and enormous memory bandwidth, the tensor cores, the split into prefill and decode, all of it is what happens when you commit to the bet that the work is wide and that nobody minds waiting.
The other thing I did not expect was how little of the chip is arithmetic. A billion transistors of multiply-adds on a die of 76 billion, and the rest is registers, caches, schedulers, and memory controllers, machinery for feeding the arithmetic rather than doing it. The last four generations spent almost their entire transistor growth on that machinery and on making each number smaller. The physics is the same physics: a switch, on or off, and the distance to the filing cabinet. The engineering is what changed.
The two parts before this one added the frame around the machine, after two earlier ones added the wafer under it and the picture it was built to draw: the watts that flow through every level of the stack and out as heat, and the other shapes of chip, the NPU in your phone and the TPU in Google's racks, that do the same arithmetic on different terms. If you came here from ML Basics, LLM Basics, or How LLMs Talk, you now have the last layer of the stack. Those series explained what a model is and how it runs. This one explained the machine it runs on, from the w × x + b of a single neuron down to the switch that computes it, and the tube that computed it first, and up to the rack that computes trillions of them a second. The next time you see a headline about a new chip, read the bandwidth line first, then the dense tensor figure at the precision that matters, and then ask what your job needs. That is all the spec sheet was ever saying.