Tensor Cores

Open the RTX 4090's spec sheet and there are two numbers for how fast it computes, and they differ by eight times. The FP32 lanes of Part 7 manage 82.6 trillion operations a second. The tensor cores, the small green box in each SM partition, manage 660 trillion at 8-bit precision, and Nvidia's headline figure of 1.3 quadrillion adds a further trick we will get to. Same chip, same clock, same power. Where does an eightfold gain come from?

The answer is two ideas stacked on top of each other, and both are things you have already seen. The first is that the operation neural networks spend nearly all their time on, matrix multiplication, has so much reuse that a purpose-built unit can wire the reuse in rather than shuffling numbers through registers. The second is the table at the end of Part 3: a multiplier's size grows with the square of its mantissa width, so a unit that works in fewer bits can contain many more multipliers. Tensor cores are where those two ideas meet, and they are the reason a graphics chip can be called an AI chip with a straight face.

The operation

LLM Basics showed that a transformer is mostly matrix multiplications, and it is worth being precise about why. A layer of a neural network takes a vector of inputs x and produces a vector of outputs, and each output is w · x + b summed over all the inputs, with its own row of weights. Write all the rows together and the whole layer is one matrix W multiplied by the vector x. Feed in a batch of tokens at once, and x becomes a matrix too. So a layer is a matrix times a matrix, and a model is hundreds of those in sequence. Attention is more matrix products. Almost nothing else is left.

A matrix multiply A × B computes every element of the result as a dot product of a row of A with a column of B. The key fact is reuse: each element of A participates in one multiply-add for every column of B, and each element of B in one for every row of A. In a 4,096 by 4,096 multiply, every number is used 4,096 times. Part 8 explained how shared memory captures that reuse so the numbers only cross from DRAM once. But there is a second, smaller bottleneck it does not fix.

Why ordinary lanes are the wrong shape

Think about what one lane does per multiply-add. It reads two or three operands out of the register file, computes, and writes one result back. For a matrix multiply, the operands it reads are numbers that its neighbouring lanes are also reading, this tick or the next, because they are working on adjacent elements of the same rows and columns. The arithmetic takes one tick. The register file traffic to deliver the operands is the limit on how many multiply-adds per tick a partition can sustain, and it is a wiring limit, not a transistor-count limit. You can put more multipliers in, but you cannot feed them.

The tensor core is an arithmetic unit shaped like the problem. Instead of one multiply-add on three scalars, its instruction takes small tiles of the matrices, a few rows of A and a few columns of B, and computes the entire tile of products in one operation, accumulating the results into a tile of C: D = A × B + C, where all four are matrices. Volta's first version in 2017 took 4 by 4 tiles, which is 64 multiply-adds per instruction per tensor core. Later generations take larger tiles and issue them faster.

Inside the unit is a grid of multipliers feeding into trees of adders, and the important part is the wiring. Each element of the A tile is delivered once and physically broadcast to every multiplier in its row. Each element of B is delivered once and broadcast down its column. The reuse that a lane would have to achieve by reading the same register over and over is built into the copper. Per multiply-add, the operand traffic drops by roughly the width of the tile, and suddenly you can feed dozens of multipliers where before you could feed one. That is the first factor. The second factor is what those multipliers are multiplying.

Fewer bits, more multipliers

Recall from Part 3 that multiplying two floating-point numbers means multiplying their mantissas, and that the size of a mantissa multiplier grows with the square of its width. An FP32 multiplier handles 24-bit mantissas. An FP16 multiplier handles 11-bit ones, and is about a fifth the size. FP8 handles 4 bits, and is a thirty-sixth. So in the area of one FP32 multiplier you can place a handful of FP16 units or dozens of FP8 ones, and they all run in the same tick on the same power.

Combine that with the tile wiring above and you get the tensor core's throughput ladder, which also happens to be the history of Nvidia's last five generations:

Each row halves the bits per number relative to the one a generation or two before it, and each halving buys roughly double the arithmetic per square millimetre and halves the bytes moved per number, which is Part 8's bottleneck relieved at the same time. Two wins for one change. No wonder the industry keeps pulling the lever.

What you lose, and why a neural network doesn't mind

The obvious objection is that a number with a 3-bit mantissa is barely a number at all. Type a value below and see what each format actually stores.

The relative error at FP8 is a few per cent, and at FP4 it can be twenty or more. That would be disastrous for accounting software. It turns out to be nearly harmless for a neural network, for a reason that goes back to ML Basics: the weights were found by gradient descent, a search that is itself noisy and approximate, and the network has to be robust to small perturbations of its weights or it would never have trained in the first place. Rounding a weight from 0.0317 to 0.03125 is a smaller change than the last few training steps made to it. The knowledge in the model is spread across billions of numbers, and none of them needs to be exact.

What a network does mind is range. Look at the "largest value" column. An FP4 number cannot exceed 6, and FP8 E4M3 stops at 448, so a layer whose values happen to run into the thousands would overflow. This is why low-precision formats never travel alone. They come with a scale factor: a single higher-precision number that says "multiply everything in this block by 512". Hopper's Transformer Engine keeps one scale per tensor and adjusts it layer by layer during training, choosing FP8 where it is safe and falling back where it is not. Blackwell goes finer, with one scale for every small block of 16 or 32 numbers, which lets FP4 work where a per-tensor scale would have failed. And the accumulation, the running sum in the adder trees, is always done in a wider format than the products, typically FP32, so that adding up thousands of small products does not lose them. The rule of thumb is: multiply cheaply, add carefully, and keep the scale factor honest.

The asterisk on the spec sheet

That 1.3 quadrillion figure for the RTX 4090 is not 660 trillion rounded up. It is the sparsity number. Since Ampere, if the weights are arranged so that in every group of four, two are zero, the tensor core can skip the zeros and run twice as fast. Nvidia calls this 2:4 structured sparsity, and it works when a model has been trained or pruned to have that pattern, which most have not. Every Nvidia spec sheet since 2020 quotes tensor performance both ways, usually with the sparse number in the headline and the dense number in a footnote. When you compare chips, compare dense to dense and the same format to the same format, or the comparison means nothing.

Read the table across a row and you see the tensor cores' advantage over the plain lanes, eight times on the 4090 and over a hundred on the B200 once FP4 is in play. Read it down a column and you see the generational gains. Read the FP4 column and you see why Blackwell's headline number is so much larger than Hopper's: a good part of it is the format, not the silicon.

Where this leaves us

A tensor core is a multiply-add unit shaped like a matrix instead of a scalar, with the reuse of matrix arithmetic built into its wiring, filled with multipliers made small by using few bits. Since 2017 it has been the part of the GPU that does the actual work of AI, and each generation's leap in AI performance has come mostly from two moves: bigger tiles fed by better memory, and fewer bits per number backed by cleverer scaling. The plain lanes of Part 7 still matter, for everything that is not a matrix multiply, but they are no longer where the transistors go.

We now have the entire inside of the chip: switches, gates, lanes, SMs, the memory that feeds them, and the tensor cores that do the heavy lifting. What we have not yet seen is how any of it gets told what to do. A GPU cannot run a program by itself. Something has to load the weights into its memory, launch the work, and collect the answer, and that something is the CPU sitting beside it on the motherboard, speaking a language called CUDA.

Next: Talking to the GPU: host and device, kernels, grids and blocks, how a dozen lines of CUDA map exactly onto the hardware of Part 7, and why the software is the real moat.