Thirty-two parts ago I opened a microwave oven, found a vacuum tube inside, and asked how we got from that to a machine with 76 billion switches that draws games and runs language models. The answer took a while, because it is a stack of ideas rather than a single trick, and each layer only makes sense once the one below it is in place. This part puts the stack back together in one picture, then hands you the skill the series was really for: picking up any GPU spec sheet and knowing what every line means and which ones matter for your job. It ends with a glossary. This is the part to bookmark.
The stack, bottom to top
The stack above is the chain of containment that the running calculation passes through. Here is the same stack drawn the other way, as block diagrams, so that the parts that are not on the multiply-add's direct path, the caches, the switches, the schedulers, the tensor and RT cores, are in the picture too. Each panel opens up the highlighted block of the one above it.
Read it upward and it is the series in order. Below the bottom box sit the relay and the vacuum tube, which did the transistor's job first and could not be made small, cool, or numerous enough to build this stack from. A transistor is a switch. A dozen switches make a gate. Thousands of gates make a unit that computes w × x + b. Wrap that in registers and you have a lane. Put 128 lanes under four schedulers, each reading one instruction to 32 lanes at once and switching between warps whenever one waits for memory, and you have an SM. Tile 144 SMs around a cache and you have a die. Join two dies and surround them with stacked memory and you have a package. Wire 72 packages together at 1.8 TB/s each and you have the machine that trains a language model. Change the names and it is AMD's stack too: compute units for SMs, wavefronts for warps, Infinity Fabric for NVLink, more pieces of silicon in each package, and the same memory wall at every level.
Read it downward and it is the memory wall. At every level the arithmetic is cheap and getting the numbers to it is the problem, and every level's design is a way of moving fewer bytes: registers instead of memory, shared memory tiles, the L2, HBM on the package, NVLink instead of PCIe, fewer bits per number in the tensor cores. If you remember one sentence from the thirty-three parts, make it the one from Part 12: compute is cheap, bytes are expensive.
How to read a spec sheet
Every line on a GPU spec sheet corresponds to something in this series. Here is the H100 SXM's, line by line, with what each one is really telling you.
| Line on the sheet | What it means | Where |
|---|---|---|
| TSMC 4N, 80 billion transistors | the process generation (a marketing name) and the switch count, which sets a hard ceiling on everything else | Part 3, 19 |
| 814 mm² die, 132 of 144 SMs enabled | the printed area, which fixes how many fit on a wafer and how many are clean, and the yield story made visible: the SMs that failed were fused off | Part 3, 19 |
| 132 SMs, 16,896 CUDA cores | 132 copies of the building block, each with 128 lanes. "CUDA cores" is lanes, not cores in the CPU sense | Part 9 |
| Boost clock 1.98 GHz | ticks per second. Multiply by lanes and by two for the FP32 peak | Part 5 |
| 528 tensor cores, 4th generation | four per SM. The generation tells you which precisions they support | Part 13 |
| FP32 67 TFLOPS | the plain lanes' peak. Rarely the number that matters for AI | Part 9 |
| FP16 989 TFLOPS, FP8 1,979 TFLOPS, "with sparsity" 2× | tensor core peaks. Always find the dense figure and compare like with like | Part 13 |
| 80 GB HBM3, 3.35 TB/s | how much model fits, and how many times a second it can be read. For generation, this is the line that matters most | Part 12 |
| 50 MB L2 | the chip-wide cache that catches reuse before it goes off-die | Part 12 |
| 228 KB shared memory per SM | the programmer-managed tile buffer. Bigger means bigger tiles for the tensor cores | Part 12 |
| NVLink 900 GB/s, PCIe Gen 5 128 GB/s | the roads to other GPUs and to the CPU. About seven to one between them | Part 14, 15 |
| Compute capability 9.0 | which CUDA features the chip has. 9.0 means clusters, TMA, FP8 | Part 14 |
| 700 W | the power budget for the SXM module. Also the cooling problem, and the datacentre's electricity bill | Part 31 |
The same lines with different numbers describe any GPU from the last decade. A gaming card adds RT cores, ROPs, texture units, video encoders, and display outputs, the pipeline hardware of Part 20 that Part 21 covered. A newer data centre part adds dies per package and a rack-level line for the NVLink domain.
An AMD sheet uses different words for many of the same lines. Here are the ones that differ on an MI355X's.
| Line on the AMD sheet | What it corresponds to | Where |
|---|---|---|
| 256 compute units, 16,384 stream processors | SMs and CUDA cores: 256 building blocks of 64 lanes each | Part 10 |
| 1,024 matrix cores | tensor cores, four per compute unit | Part 10, 13 |
| 8 XCDs on 2 IODs, 185 billion transistors | dies per package: compute dies stacked on base dies | Part 19, 28 |
| 256 MB AMD Infinity Cache | no Nvidia equivalent: an extra cache level in front of the HBM | Part 12, 22 |
| 288 GB HBM3E, 8 TB/s | the same line, read the same way | Part 12 |
| MXFP4 and MXFP6 10 PFLOPS, "with sparsity" 2× | the same dense-or-sparse question, plus six-bit at the four-bit rate | Part 13, 28 |
| 7 Infinity Fabric links, 1,075 GB/s | NVLink, but wired GPU to GPU without a switch, so one pair gets a seventh | Part 15 |
| OAM module, 1,400 W | the SXM line: the module form factor and its power budget | Part 31 |
| LLVM target gfx950, ROCm 7 | compute capability: which features the compiler can use, and the software stack | Part 14 |
Three questions for any chip
When someone asks whether a given GPU is a good fit for a given job, the spec sheet reduces to three questions, and they are the roofline of Part 12 in words.
How many multiply-adds per second can it do, at the precision the job will actually use, without sparsity? That is the flat roof.
How many bytes per second can it move, from its own memory for one GPU, and from its neighbours over NVLink if the model is split? That is the sloping roof.
How many operations does the job do per byte it moves? A chatbot generating text does about one per byte at 16-bit precision. A training run or a long-prompt prefill does thousands. A single-user workload sits far left and is limited entirely by the second question, and the fix is fewer bytes: quantise, batch, or buy bandwidth. A batched or training workload sits right and is limited by the first, and the fix is more tensor cores or fewer bits per number.
Everything else on the sheet is detail that adjusts one of those three numbers. That is the whole method, and it is the one I now use before I read any benchmark.
Seven chips on one page
| RTX 4090 (Ada, 2022) | RTX 5090 (Blackwell, 2025) | H100 SXM (Hopper, 2022) | B200 (Blackwell, 2024) | Rubin (2026, stated) | MI355X (CDNA 4, 2025) | MI455X (CDNA 5, 2026, stated) | |
|---|---|---|---|---|---|---|---|
| Transistors | 76.3 billion | 92.2 billion | 80 billion | 208 billion, 2 dies | 336 billion, 2 dies | 185 billion, 10 dies | 320 billion, 12 dies |
| SMs or compute units | 128 | 170 | 132 | not published | 224 | 256 CUs | 256 WGPs |
| Memory | 24 GB GDDR6X | 32 GB GDDR7 | 80 GB HBM3 | up to 192 GB HBM3e | 288 GB HBM4 | 288 GB HBM3E | 432 GB HBM4 |
| Bandwidth | 1.0 TB/s | 1.79 TB/s | 3.35 TB/s | 8 TB/s | 22 TB/s target | 8 TB/s | 23.3 TB/s |
| FP8 tensor, dense | 660 TFLOPS | about 830 TFLOPS | 1,979 TFLOPS | 4.5 PFLOPS | 17.5 PFLOPS | 5 PFLOPS | 20 PFLOPS |
| FP4 tensor | no | about 1.7 PFLOPS dense | no | 9 PFLOPS dense | 50 PFLOPS, stated | 10 PFLOPS dense | 40 PFLOPS dense, stated |
| GPU-to-GPU link | PCIe only | PCIe only | 900 GB/s | 1.8 TB/s | 3.6 TB/s | 1,075 GB/s, no switch | 3.6 TB/s |
| Power | 450 W | 575 W | 700 W | 1,000 to 1,200 W | not published | 1,400 W | not published |
The RTX 5090's tensor figures are derived from Nvidia's "AI TOPS" number, which is an FP4 figure quoted with sparsity (Part 32 explains TOPS), and should be read as approximate. The Rubin column is Nvidia's own preliminary figures, as Part 29 explained, and its 50 PFLOPS of FP4 is a figure with sparsity; the dense equivalent is 35. The two AMD columns are AMD's own figures from Part 28 and Part 30, and the MI455X's, like Rubin's, have not yet been measured by anyone else. Look at the bandwidth row and then the FP4 row, and you are looking at what the last four years of GPU design were about.
Glossary
| Term | Meaning |
|---|---|
| Relay | an electromagnet that moves a contact. The oldest electrically controlled switch, slow and big, still used wherever big currents need switching with total isolation. (Part 1) |
| Vacuum tube | a heated cathode, a grid, and a plate in a vacuum. The first amplifier and the switch of the first computers, limited by heat, size, and burnout. Still used for extreme power and frequency. (Part 2) |
| Tyranny of numbers | the 1950s wall where every extra tube or soldered joint added cost, heat, and failures. Solved by the transistor and then the integrated circuit. (Part 2) |
| Transistor | a switch flipped by voltage on its gate. The only component in a chip. (Part 3) |
| Wafer | the 300 mm disc of pure silicon on which chips are printed, the unit the whole industry counts in. (Part 3, 19) |
| Die | one printed copy of a chip on the wafer, cut out and packaged. The thing whose area and transistor count a spec sheet quotes. (Part 3, 19) |
| Yield, binning | the share of dies that come out clean, and the sorting of the rest into products with units fused off. Why the 4090 has 128 of 144 SMs. (Part 19) |
| Reticle limit | the largest die a lithography machine can print in one exposure, 858 mm². Why Blackwell has two dies. (Part 3, 19, 27) |
| Package, interposer, CoWoS | the carrier that connects a die to the board, the slab of silicon wiring that lets HBM sit beside the GPU, and TSMC's name for the combination. (Part 19) |
| HBM stack, TSV | eight or twelve memory dies stacked and joined by holes through the silicon, so a fingernail of DRAM offers a thousand wires. (Part 12, 19) |
| Frame, frame buffer | one full screen of pixel colours, and the memory that holds it. (Part 20) |
| Vertex, triangle, shader | a corner with a position and a few attributes, the only shape a GPU draws, and a small program run once per vertex or per pixel. (Part 20) |
| Rasteriser, ROP, texture unit | the fixed-function units that find the pixels a triangle covers, write finished pixels, and sample pictures onto surfaces. (Part 20) |
| Dennard scaling | the rule, broken around 2005, that smaller transistors ran faster for the same power. Its end is why chips went wide instead of fast. (Part 3) |
| Logic gate | a few transistors implementing a truth table: AND, OR, XOR, NOT, NAND. (Part 4) |
| FMA | fused multiply-add, a × b + c in one operation with one rounding. The atom of GPU arithmetic. (Part 4) |
| FP32, FP16, BF16, FP8, FP4 | floating-point formats named by total bits. Fewer bits means a smaller multiplier and fewer bytes to move. (Part 4, 13) |
| Register | a small bank of SRAM next to the arithmetic, reachable in one tick. (Part 5) |
| Clock | the signal that ticks every register at once. Its rate is capped by the slowest chain of gates. (Part 5) |
| Latency, bandwidth | how long one access takes, and how much data arrives per second. Independent, and only the second is easy to improve. (Part 7) |
| Cache, L1, L2 | copies of recently used memory kept close by. GPUs have small L1s and, recently, large L2s. (Part 7, 12) |
| Memory wall | the growing gap between how fast a processor can compute and how fast memory can feed it. (Part 7) |
| Latency-oriented, throughput-oriented | the CPU's and GPU's opposite strategies for the memory wall: never let one thread wait, or always have another thread ready. (Part 7, 8) |
| Embarrassingly parallel | work made of independent items, like pixels or a layer of neurons. What GPUs are for. (Part 8) |
| Amdahl's law | the sequential fraction of a job caps its speedup no matter how many lanes you add. (Part 8) |
| SIMT | single instruction, multiple threads. One instruction issued to 32 lanes, each with its own data. (Part 8, 9) |
| Warp | 32 threads executed in lockstep. The unit the scheduler works in, unchanged since 2006. (Part 9) |
| SM | streaming multiprocessor. Four schedulers, 128 lanes, tensor cores, shared memory. The building block. (Part 9) |
| CUDA core | Nvidia's name for one lane. Not a core in the CPU sense. (Part 8, 9) |
| Occupancy | how many warps an SM has resident to switch between. Low occupancy means memory stalls show through. (Part 9) |
| Branch divergence | threads of a warp disagreeing at an if, forcing both paths to run with lanes masked off. (Part 9) |
| Compute unit (CU), stream processor | AMD's building block, its counterpart to the SM, and AMD's name for one lane. (Part 10) |
| Wavefront, wave32, wave64 | AMD's word for a warp: 64 threads on GCN and CDNA, 32 or 64 on RDNA. (Part 10) |
| Scalar unit | the part of an AMD compute unit that works once per wavefront rather than once per thread: loop counters, addresses, branch decisions. (Part 10) |
| WGP, local data share | RDNA's work-group processor, a pair of compute units sharing their caches, and AMD's name for shared memory. (Part 10) |
| Chiplet, Infinity Cache | a GPU built from several dies in one package, and AMD's large last-level cache in front of memory. (Part 10, 19, 22) |
| RDNA, CDNA | AMD's two GPU architectures from 2020, for graphics and for computing, converging again from 2026. (Part 10, 22, 30) |
| XCD, I/O die | the compute dies of an AMD Instinct chip and the base dies they are stacked on. (Part 10, 25, 28) |
| Matrix core, MFMA | AMD's tensor core, and the matrix instructions a whole wavefront executes on it. (Part 10, 13) |
| HIP, ROCm | AMD's CUDA-like programming language and the open-source software stack around it. (Part 14) |
| Infinity Fabric | AMD's links between dies, between GPUs, and between CPU and GPU. (Part 15, 25) |
| OAM | the open module form factor for data-centre accelerators, AMD's equivalent of SXM. (Part 15, 31) |
| APU | one package with CPU cores and GPU compute sharing a single memory, such as the MI300A. (Part 25) |
| Hybrid bonding | stacking dies copper to copper without solder, at a pitch of a few micrometres. (Part 19) |
| MX formats | microscaling FP8, FP6 and FP4, with one shared scale for each block of 32 numbers. (Part 27, 28) |
| UALink | an open standard for joining GPUs into one scale-up domain, carried over Ethernet in AMD's Helios rack. (Part 15, 30) |
| Xe-core, XVE | Intel's building block and its vector engines, each issuing to 8 or 16 lanes. (Part 11) |
| Hardware thread, sub-group | Intel's word for a warp, 8, 16 or 32 work-items wide, the width chosen by the compiler. (Part 11, 14) |
| XMX, DPAS | Intel's matrix engines and the instruction that drives them. (Part 11, 13) |
| Shared local memory (SLM) | Intel's name for shared memory. (Part 11) |
| Render slice | Intel's group of four Xe-cores with their graphics hardware, its equivalent of a GPC. (Part 11) |
| SYCL, oneAPI, Level Zero | the open C++ standard Intel programs its GPUs in, its toolkit, and its low-level driver interface. (Part 14) |
| EMIB, Foveros | Intel's packaging: a silicon bridge buried under two neighbouring dies, and dies stacked face to face. (Part 19, 26) |
| LPDDR | the low-power memory of phones and laptops, used for very large, slower GPU memories. (Part 12, 22, 26) |
| Shared memory | per-SM SRAM the program manages directly, used to hold tiles for reuse. (Part 12) |
| GDDR, HBM | graphics memory on the board, or stacked memory on the package. Cheap and wire-limited, or expensive and very wide. (Part 12) |
| Arithmetic intensity | operations per byte moved. The one number that says whether a job is memory-bound or compute-bound. (Part 12) |
| Roofline | the diagram of peak compute (flat) and bandwidth (sloped) that shows a job's ceiling at its intensity. (Part 12) |
| Coalescing | a warp's 32 adjacent loads served by one memory transaction. Scattered loads cost up to 32. (Part 12) |
| Tensor core | a unit that multiplies small matrix tiles in one instruction, with reuse built into its wiring. (Part 13) |
| Sparsity, 2:4 | skipping zeros when two of every four weights are zero. Doubles the headline number, rarely applies. (Part 13) |
| Transformer Engine | Hopper's hardware and software for choosing scale factors so that FP8, and on Blackwell FP4, can be used safely. (Part 13, 24, 27) |
| Host, device | the CPU and its memory, the GPU and its memory. (Part 14) |
| Kernel, grid, block | a function written for one thread, launched across a grid of blocks, each block living on one SM. (Part 14) |
| PCIe | the slot that connects host and device, 32 to 64 GB/s. The slow road. (Part 14) |
| PTX, compute capability | the portable intermediate code CUDA compiles to, and the version number of a GPU's feature set. (Part 14) |
| NVLink, NVSwitch | the dedicated GPU-to-GPU link, and the switch that lets every GPU in a node or rack reach every other. (Part 15) |
| NVLink domain | the set of GPUs joined by NVSwitch, currently 72, that software can treat as one memory. (Part 15, 27) |
| Data, tensor, pipeline parallelism | the three ways to cut a model across GPUs, from least to most communication. (Part 15) |
| Superchip, NVLink-C2C | a CPU and GPU on one board joined by a coherent link, so the GPU can spill into CPU memory. (Part 15) |
| Thread block cluster, TMA | Hopper's additions: blocks that share each other's shared memory, and a unit that moves tiles without lane instructions. (Part 24) |
| NV-HBI | the 10 TB/s seam joining Blackwell's two dies into one GPU. (Part 27) |
| Prefill, decode | the compute-bound prompt-reading phase and the memory-bound generating phase of an LLM request. (Part 12, 29) |
| Disaggregated inference | running prefill and decode on different machines, each with the memory it needs. (Part 29) |
| TDP, TGP, SXM | the power a chip is designed to shed continuously, Nvidia's name for it on gaming cards, and the module form factor that allows 700 W where a PCIe card allows 350. (Part 31) |
| Dynamic and static power | the energy of flipping switches, and the leakage through switches that are off. Both end up as heat. (Part 31) |
| PUE | power usage effectiveness: the whole building's draw divided by the computers' draw. Industry average about 1.56, the best about 1.1. (Part 31) |
| WUE, withdrawal, consumption | water usage effectiveness: litres of water consumed on site per kWh the computers use. Water that evaporates in a cooling tower is consumed, water returned to the river is only withdrawn. (Part 31) |
| Direct-to-chip liquid cooling | coolant piped through a cold plate on each chip, needed once a rack passes a few tens of kilowatts. (Part 31) |
| SoC, unified memory | a phone or laptop's CPU, GPU and NPU on one die sharing one pool of memory. (Part 32) |
| NPU | neural processing unit: a fixed-function matrix engine on a system-on-chip, for running trained networks at a few watts. (Part 32) |
| Systolic array | a grid of multiply-add cells through which inputs and partial sums flow, each number fetched once. The heart of an NPU and of the TPU. (Part 32) |
| TOPS | trillions of operations per second, usually at 8-bit integer precision. Always ask how many bits, dense or sparse, and at what power. (Part 32) |
| TPU | Google's tensor processing unit: a systolic array grown to data-centre scale, with HBM and an interconnect around it. (Part 32) |
What I took from the trip
Jon Stokes's book gave me the calculator and the file clerk, and for years that was my whole model of a processor. What this series added, for me, is one more figure in the room: the clerk who reads each instruction aloud to 32 calculators at once, and keeps a dozen rows of them on the go so that whenever one row is waiting for the library, another is working. Everything distinctive about a GPU is that figure. The warp, the register file the size of a CPU's cache, the small caches and enormous memory bandwidth, the tensor cores, the split into prefill and decode, all of it is what happens when you commit to the bet that the work is wide and that nobody minds waiting.
The other thing I did not expect was how little of the chip is arithmetic. A billion transistors of multiply-adds on a die of 76 billion, and the rest is registers, caches, schedulers, and memory controllers, machinery for feeding the arithmetic rather than doing it. The last four generations spent almost their entire transistor growth on that machinery and on making each number smaller. The physics is the same physics: a switch, on or off, and the distance to the filing cabinet. The engineering is what changed.
Adding AMD's half of the story changed the picture less than I expected and more than I wanted. Less, because the second company arrived at almost every idea in this series too: the lanes in lockstep groups, the big caches, the matrix units, the stacked memory, the rack. More, because the order was different. AMD had chiplets, HBM and large caches first and the software last, Nvidia the other way round, and a good deal of what the industry looks like in 2026 follows from that.
Intel's third of the story points the same way from a different angle. It has shipped more GPUs than either rival, built the most elaborate chiplet package in this series and one of the three exascale machines of 2024, and it still found the big GPU the hardest product in computing to break into. If one lesson runs through all three companies, it is that the silicon is the part that can be bought, and the software, the systems and the years of persistence are the part that cannot.
The two parts before this one added the frame around the machine, after two earlier ones added the wafer under it and the picture it was built to draw: the watts that flow through every level of the stack and out as heat, and the other shapes of chip, the NPU in your phone and the TPU in Google's racks, that do the same arithmetic on different terms. If you came here from ML Basics, LLM Basics, or How LLMs Talk, you now have the last layer of the stack. Those series explained what a model is and how it runs. This one explained the machine it runs on, from the w × x + b of a single neuron down to the switch that computes it, and the tube that computed it first, and up to the rack that computes trillions of them a second. The next time you see a headline about a new chip, read the bandwidth line first, then the dense tensor figure at the precision that matters, and then ask what your job needs. That is all the spec sheet was ever saying.