Sixteen parts ago I opened a microwave oven, found a vacuum tube inside, and asked how we got from that to a machine with 76 billion switches that draws games and runs language models. The answer took a while, because it is a stack of ideas rather than a single trick, and each layer only makes sense once the one below it is in place. This part puts the stack back together in one picture, then hands you the skill the series was really for: picking up any GPU spec sheet and knowing what every line means and which ones matter for your job. It ends with a glossary. This is the part to bookmark.
The stack, bottom to top
Read it upward and it is the series in order. Below the bottom box sit the relay and the vacuum tube, which did the transistor's job first and could not be made small, cool, or numerous enough to build this stack from. A transistor is a switch. A dozen switches make a gate. Thousands of gates make a unit that computes w × x + b. Wrap that in registers and you have a lane. Put 128 lanes under four schedulers, each reading one instruction to 32 lanes at once and switching between warps whenever one waits for memory, and you have an SM. Tile 144 SMs around a cache and you have a die. Join two dies and surround them with stacked memory and you have a package. Wire 72 packages together at 1.8 TB/s each and you have the machine that trains a language model.
Read it downward and it is the memory wall. At every level the arithmetic is cheap and getting the numbers to it is the problem, and every level's design is a way of moving fewer bytes: registers instead of memory, shared memory tiles, the L2, HBM on the package, NVLink instead of PCIe, fewer bits per number in the tensor cores. If you remember one sentence from the seventeen parts, make it the one from Part 8: compute is cheap, bytes are expensive.
How to read a spec sheet
Every line on a GPU spec sheet corresponds to something in this series. Here is the H100 SXM's, line by line, with what each one is really telling you.
| Line on the sheet | What it means | Where |
|---|---|---|
| TSMC 4N, 80 billion transistors | the process generation (a marketing name) and the switch count, which sets a hard ceiling on everything else | Part 2 |
| 132 SMs, 16,896 CUDA cores | 132 copies of the building block, each with 128 lanes. "CUDA cores" is lanes, not cores in the CPU sense | Part 7 |
| Boost clock 1.98 GHz | ticks per second. Multiply by lanes and by two for the FP32 peak | Part 4 |
| 528 tensor cores, 4th generation | four per SM. The generation tells you which precisions they support | Part 9 |
| FP32 67 TFLOPS | the plain lanes' peak. Rarely the number that matters for AI | Part 7 |
| FP16 989 TFLOPS, FP8 1,979 TFLOPS, "with sparsity" 2× | tensor core peaks. Always find the dense figure and compare like with like | Part 9 |
| 80 GB HBM3, 3.35 TB/s | how much model fits, and how many times a second it can be read. For generation, this is the line that matters most | Part 8 |
| 50 MB L2 | the chip-wide cache that catches reuse before it goes off-die | Part 8 |
| 228 KB shared memory per SM | the programmer-managed tile buffer. Bigger means bigger tiles for the tensor cores | Part 8 |
| NVLink 900 GB/s, PCIe Gen 5 128 GB/s | the roads to other GPUs and to the CPU. About seven to one between them | Part 10, 11 |
| Compute capability 9.0 | which CUDA features the chip has. 9.0 means clusters, TMA, FP8 | Part 10 |
| 700 W | the power budget. Also the cooling problem, and the datacentre's electricity bill | Part 2 |
The same lines with different numbers describe any GPU from the last decade. A gaming card adds RT cores, video encoders, and display outputs, all of which Part 13 covered. A newer data centre part adds dies per package and a rack-level line for the NVLink domain.
Three questions for any chip
When someone asks whether a given GPU is a good fit for a given job, the spec sheet reduces to three questions, and they are the roofline of Part 8 in words.
How many multiply-adds per second can it do, at the precision the job will actually use, without sparsity? That is the flat roof.
How many bytes per second can it move, from its own memory for one GPU, and from its neighbours over NVLink if the model is split? That is the sloping roof.
How many operations does the job do per byte it moves? A chatbot generating text does about one per byte at 16-bit precision. A training run or a long-prompt prefill does thousands. A single-user workload sits far left and is limited entirely by the second question, and the fix is fewer bytes: quantise, batch, or buy bandwidth. A batched or training workload sits right and is limited by the first, and the fix is more tensor cores or fewer bits per number.
Everything else on the sheet is detail that adjusts one of those three numbers. That is the whole method, and it is the one I now use before I read any benchmark.
Five chips on one page
| RTX 4090 (Ada, 2022) | RTX 5090 (Blackwell, 2025) | H100 SXM (Hopper, 2022) | B200 (Blackwell, 2024) | Rubin (2026, stated) | |
|---|---|---|---|---|---|
| Transistors | 76.3 billion | 92.2 billion | 80 billion | 208 billion, 2 dies | 336 billion, 2 dies |
| SMs | 128 | 170 | 132 | not published | 224 |
| Memory | 24 GB GDDR6X | 32 GB GDDR7 | 80 GB HBM3 | up to 192 GB HBM3e | 288 GB HBM4 |
| Bandwidth | 1.0 TB/s | 1.79 TB/s | 3.35 TB/s | 8 TB/s | 22 TB/s target |
| FP8 tensor, dense | 660 TFLOPS | about 830 TFLOPS | 1,979 TFLOPS | 4.5 PFLOPS | 17.5 PFLOPS |
| FP4 tensor | no | about 1.7 PFLOPS dense | no | 9 PFLOPS dense | 50 PFLOPS, stated |
| GPU-to-GPU link | PCIe only | PCIe only | 900 GB/s | 1.8 TB/s | 3.6 TB/s |
| Power | 450 W | 575 W | 700 W | 1,000 to 1,200 W | not published |
The RTX 5090's tensor figures are derived from Nvidia's "AI TOPS" number, which is quoted with sparsity, and should be read as approximate. The Rubin column is Nvidia's own preliminary figures, as Part 16 explained. Look at the bandwidth row and then the FP4 row, and you are looking at what the last four years of GPU design were about.
Glossary
| Term | Meaning |
|---|---|
| Relay | an electromagnet that moves a contact. The oldest electrically controlled switch, slow and big, still used wherever big currents need switching with total isolation. (Part 1) |
| Vacuum tube | a heated cathode, a grid, and a plate in a vacuum. The first amplifier and the switch of the first computers, limited by heat, size, and burnout. Still used for extreme power and frequency. (Part 1) |
| Tyranny of numbers | the 1950s wall where every extra tube or soldered joint added cost, heat, and failures. Solved by the transistor and then the integrated circuit. (Part 1) |
| Transistor | a switch flipped by voltage on its gate. The only component in a chip. (Part 2) |
| Reticle limit | the largest die a lithography machine can print in one exposure, about 850 mm². Why Blackwell has two dies. (Part 2, 15) |
| Dennard scaling | the rule, broken around 2005, that smaller transistors ran faster for the same power. Its end is why chips went wide instead of fast. (Part 2) |
| Logic gate | a few transistors implementing a truth table: AND, OR, XOR, NOT, NAND. (Part 3) |
| FMA | fused multiply-add, a × b + c in one operation with one rounding. The atom of GPU arithmetic. (Part 3) |
| FP32, FP16, BF16, FP8, FP4 | floating-point formats named by total bits. Fewer bits means a smaller multiplier and fewer bytes to move. (Part 3, 9) |
| Register | a small bank of SRAM next to the arithmetic, reachable in one tick. (Part 4) |
| Clock | the signal that ticks every register at once. Its rate is capped by the slowest chain of gates. (Part 4) |
| Latency, bandwidth | how long one access takes, and how much data arrives per second. Independent, and only the second is easy to improve. (Part 5) |
| Cache, L1, L2 | copies of recently used memory kept close by. GPUs have small L1s and, recently, large L2s. (Part 5, 8) |
| Memory wall | the growing gap between how fast a processor can compute and how fast memory can feed it. (Part 5) |
| Latency-oriented, throughput-oriented | the CPU's and GPU's opposite strategies for the memory wall: never let one thread wait, or always have another thread ready. (Part 5, 6) |
| Embarrassingly parallel | work made of independent items, like pixels or a layer of neurons. What GPUs are for. (Part 6) |
| Amdahl's law | the sequential fraction of a job caps its speedup no matter how many lanes you add. (Part 6) |
| SIMT | single instruction, multiple threads. One instruction issued to 32 lanes, each with its own data. (Part 6, 7) |
| Warp | 32 threads executed in lockstep. The unit the scheduler works in, unchanged since 2006. (Part 7) |
| SM | streaming multiprocessor. Four schedulers, 128 lanes, tensor cores, shared memory. The building block. (Part 7) |
| CUDA core | Nvidia's name for one lane. Not a core in the CPU sense. (Part 6, 7) |
| Occupancy | how many warps an SM has resident to switch between. Low occupancy means memory stalls show through. (Part 7) |
| Branch divergence | threads of a warp disagreeing at an if, forcing both paths to run with lanes masked off. (Part 7) |
| Shared memory | per-SM SRAM the program manages directly, used to hold tiles for reuse. (Part 8) |
| GDDR, HBM | graphics memory on the board, or stacked memory on the package. Cheap and wire-limited, or expensive and very wide. (Part 8) |
| Arithmetic intensity | operations per byte moved. The one number that says whether a job is memory-bound or compute-bound. (Part 8) |
| Roofline | the diagram of peak compute (flat) and bandwidth (sloped) that shows a job's ceiling at its intensity. (Part 8) |
| Coalescing | a warp's 32 adjacent loads served by one memory transaction. Scattered loads cost up to 32. (Part 8) |
| Tensor core | a unit that multiplies small matrix tiles in one instruction, with reuse built into its wiring. (Part 9) |
| Sparsity, 2:4 | skipping zeros when two of every four weights are zero. Doubles the headline number, rarely applies. (Part 9) |
| Transformer Engine | Hopper's hardware and software for choosing scale factors so that FP8, and on Blackwell FP4, can be used safely. (Part 9, 14, 15) |
| Host, device | the CPU and its memory, the GPU and its memory. (Part 10) |
| Kernel, grid, block | a function written for one thread, launched across a grid of blocks, each block living on one SM. (Part 10) |
| PCIe | the slot that connects host and device, 32 to 64 GB/s. The slow road. (Part 10) |
| PTX, compute capability | the portable intermediate code CUDA compiles to, and the version number of a GPU's feature set. (Part 10) |
| NVLink, NVSwitch | the dedicated GPU-to-GPU link, and the switch that lets every GPU in a node or rack reach every other. (Part 11) |
| NVLink domain | the set of GPUs joined by NVSwitch, currently 72, that software can treat as one memory. (Part 11, 15) |
| Data, tensor, pipeline parallelism | the three ways to cut a model across GPUs, from least to most communication. (Part 11) |
| Superchip, NVLink-C2C | a CPU and GPU on one board joined by a coherent link, so the GPU can spill into CPU memory. (Part 11) |
| Thread block cluster, TMA | Hopper's additions: blocks that share each other's shared memory, and a unit that moves tiles without lane instructions. (Part 14) |
| NV-HBI | the 10 TB/s seam joining Blackwell's two dies into one GPU. (Part 15) |
| Prefill, decode | the compute-bound prompt-reading phase and the memory-bound generating phase of an LLM request. (Part 8, 16) |
| Disaggregated inference | running prefill and decode on different machines, each with the memory it needs. (Part 16) |
What I took from the trip
Jon Stokes's book gave me the calculator and the file clerk, and for years that was my whole model of a processor. What this series added, for me, is one more figure in the room: the clerk who reads each instruction aloud to 32 calculators at once, and keeps a dozen rows of them on the go so that whenever one row is waiting for the library, another is working. Everything distinctive about a GPU is that figure. The warp, the register file the size of a CPU's cache, the small caches and enormous memory bandwidth, the tensor cores, the split into prefill and decode, all of it is what happens when you commit to the bet that the work is wide and that nobody minds waiting.
The other thing I did not expect was how little of the chip is arithmetic. A billion transistors of multiply-adds on a die of 76 billion, and the rest is registers, caches, schedulers, and memory controllers, machinery for feeding the arithmetic rather than doing it. The last four generations spent almost their entire transistor growth on that machinery and on making each number smaller. The physics is the same physics: a switch, on or off, and the distance to the filing cabinet. The engineering is what changed.
If you came here from ML Basics, LLM Basics, or How LLMs Talk, you now have the last layer of the stack. Those series explained what a model is and how it runs. This one explained the machine it runs on, from the w × x + b of a single neuron down to the switch that computes it, and the tube that computed it first, and up to the rack that computes trillions of them a second. The next time you see a headline about a new chip, read the bandwidth line first, then the dense tensor figure at the precision that matters, and then ask what your job needs. That is all the spec sheet was ever saying.