Many GPUs as One

Take a large open model, say one with 405 billion parameters, and store it at 16 bits per weight. That is 810 GB. The biggest GPU memory on the market as I write this is 288 GB. At 8 bits it is still 405 GB. There is no GPU it fits on, and there is not going to be one, because models have grown faster than memory for a decade and show no sign of stopping.

So the model is cut up. Some layers go on this GPU, some on that one, or each layer's matrices are sliced so that every GPU holds a strip of every weight. Either way, generating a single token now requires the GPUs to exchange partial results, over and over, hundreds of times per token. And that changes the question of this series. Up to now it has been "how does a GPU compute". From here it becomes "how do many GPUs behave as one", and the answer is a ladder of interconnects that makes the memory hierarchy of Part 8 look short.

PCIe is not enough

Part 10 ended with the road between host and device: PCIe, 64 GB/s each way on a Gen 5 slot. If two GPUs share a machine and want to exchange data through that road, it goes up through the CPU and back down again, and 64 GB/s is the ceiling. Against HBM's 3 to 8 TB/s, that is fifty to a hundred times too slow. A layer's activations for a large batch can be hundreds of megabytes, and sending them over PCIe takes milliseconds while the matrix multiply that produced them took microseconds. Tensor cores would spend their lives waiting on the bus.

The fix is a dedicated link between GPUs that never touches PCIe or the CPU. Nvidia's is NVLink, introduced with Pascal in 2016, and its growth is the cleanest illustration of how much the industry has bet on multi-GPU work:

Those are totals across all of a GPU's links, counting both directions, which is how Nvidia quotes them. Counted the same way, an H100's PCIe slot manages 128 GB/s, so NVLink is seven times wider. It is still a quarter of the HBM bandwidth, which is the right way to think about NVLink: fast enough that a well-designed program can overlap the talking with the computing, not fast enough to be ignored.

Everyone talks to everyone

A link joins two GPUs. With eight GPUs in a server, connecting every pair directly needs seven links per GPU, 28 in all, and each pair then gets only a seventh of the GPU's bandwidth. Add more GPUs and the count grows with the square. Early multi-GPU boards compromised, with each GPU having fast links to some neighbours and slower paths to the rest, and programmers had to care which GPU held what. Nvidia's answer, from Volta onward, was the NVSwitch: a chip that does for NVLink what an Ethernet switch does for a network, so that every GPU can talk to every other at full NVLink speed at the same time. A board with eight GPUs and a set of NVSwitches is called an HGX, and Nvidia's own server built around one is the DGX. An eight-GPU HGX H100 node has 640 GB of HBM and every GPU in it can reach every other at 900 GB/s. For years, that eight-GPU box was the atom of AI infrastructure.

Blackwell moved the boundary. The GB200 NVL72 is a full rack, 72 Blackwell GPUs and 36 Grace CPUs across 18 compute trays, with 9 trays of NVSwitch chips wired so that all 72 GPUs form one NVLink domain with 130 TB/s of total bandwidth between them. That is 13.4 TB of HBM in a single addressable pool. A model that needs 800 GB fits with room to spare, and to the software, the rack looks much like one enormous GPU, with the caveat that reaching a neighbour's memory over NVLink is still several times slower than reaching your own. The rack draws around 120 kW, roughly what a hundred homes use, and is cooled by liquid piped through every tray, because there is no way to move that much heat with air. Vera Rubin's rack keeps the same 72-GPU shape with everything doubled, and Part 16 has the numbers.

The ladder, now with rungs below the die

Beyond one rack, the next rung down is the ordinary datacentre network, InfiniBand or Ethernet, with a network interface card (NIC) per GPU running at 400 or 800 gigabits a second, which is 50 to 100 gigabytes. Another tenfold drop. Training a frontier model uses tens of thousands of GPUs, so most pairs of GPUs in such a system are connected through this bottom rung, and the art of distributed training is arranging the work so that the heavy traffic stays on the fast rungs.

Each rung is several times slower than the one above it. The ladder from Part 8 had four rungs, registers, shared memory, L2, and HBM. This part adds four more below them: NVLink, the CPU link, the network, and PCIe. A number can live at any of eight levels, and the entire craft of running large models is keeping each number as high up the ladder as its use allows.

Three ways to cut a model

How the work is split across GPUs determines which rung carries the traffic, and there are three basic cuts, usually combined.

Data parallelism gives every GPU a complete copy of the model and a different slice of the data. During training, each GPU computes gradients on its slice and then all of them average their gradients before updating, an operation called an all-reduce that touches every weight in the model. It happens once per training step, which is rarely enough that it can go over the network. It also requires the whole model to fit on one GPU, which for large models it does not.

Tensor parallelism slices each matrix across several GPUs, so that each holds a strip of every weight and computes a strip of every result. The strips have to be combined at every layer, hundreds of times per token, so this cut needs the fastest rung available. It stays within an NVLink domain, which is the reason the domain exists and the reason it keeps growing.

Pipeline parallelism puts the first layers on one GPU, the next layers on another, and so on, passing activations from stage to stage. Only the layer boundaries need to communicate, so it tolerates slower links, and a large training run typically uses tensor parallelism inside each node or rack, pipeline parallelism across racks, and data parallelism across everything. The collective operations underneath all of this, the all-reduces and all-gathers, come from a library called NCCL, one more storey of the software stack from Part 10.

The superchip

There is one more link to name, and it changes the relationship from Part 10. A Grace CPU is Nvidia's own processor: 72 Arm cores with up to 480 GB of ordinary LPDDR5X memory. On a GH200 or GB200 board it is joined to the GPU not by PCIe but by NVLink-C2C (chip-to-chip), a 900 GB/s link that also makes the two memories coherent: the GPU can read the CPU's memory directly, as if it were a slower part of its own. Nvidia calls the pair a superchip. The point is capacity. HBM is fast and scarce, LPDDR5X is slow and plentiful, and with a coherent link a model can keep its most-used weights in HBM and let the rest, or the growing conversation cache of a long chat, spill into the CPU's half-terabyte at a speed only a few times below HBM rather than fifty. GB200 puts one Grace beside two Blackwells, and the NVL72's 36 Grace CPUs contribute 17 TB of LPDDR5X to the rack.

Where this leaves us

The unit of "a GPU" has been growing throughout this series: a lane, an SM of 128 lanes, a die of 144 SMs, then, from Blackwell on, a package of two dies, a node of eight packages, and now a rack of 72 in one NVLink domain, drawing 120 kW and cooled by liquid. Nvidia designs and sells the rack. It is, in every sense that matters to the software, one machine.

The growth was forced, not chosen. Models outgrew a die, then a package, then a node, and each time the interconnect had to become part of the computer rather than a cable between computers. Keep the ladder in mind. When Part 15 says NVLink 5 doubled to 1.8 TB/s, and Part 16 says NVLink 6 doubled again, you will know why those numbers get top billing next to the tensor core figures.

We have now assembled the whole machine from the switch up. The last six parts look at how it was actually built, generation by generation: first a short history of how Nvidia got from a graphics chip to the A100, and then one part each for Ada, Hopper, Blackwell, and Vera Rubin.

Next: A Short History: from the G80 that first let a graphics card run C, through Fermi, Kepler, Maxwell, Pascal, Volta, Turing, and Ampere, and the three eras those chips belong to.