Every Nvidia architecture is named after a scientist, and if you list them in order, Tesla, Fermi, Kepler, Maxwell, Pascal, Volta, Turing, Ampere, Ada Lovelace, Hopper, Blackwell, Rubin, you have the whole story of how a company that made chips for video games ended up building the machines that train language models. The first eleven parts of this series built a GPU from the switch up. The last six look at the real ones, and this part is the run-up: eight generations in fourteen years, and the three eras they fall into.
I will not pretend every generation was a revolution. Most were refinements, and some were mainly about power. But four of them changed what a GPU is, and I want you to be able to spot which.
Era one: a graphics chip you could program (2006 to 2012)
Before 2006 a graphics chip was a pipeline of specialised stages, one kind of unit for transforming vertices, another for colouring pixels, wired in a fixed order. The G80, sold as the GeForce 8800 GTX in November 2006, threw that away. It had 128 identical units, called stream processors, that could do any stage's work, scheduled dynamically. Nvidia called the design "unified shaders", and it is the SM of Part 7 in embryo: a general-purpose lane, a scheduler in front of it, and the warp of 32 threads that has not changed since. G80 had 681 million transistors on a 90 nm process. Three months later Nvidia released CUDA, and for the first time you could write a C function and run it on a graphics card without disguising it as a picture. That was the first accident of history: a decision to make the hardware general, taken for graphics reasons, that turned out to matter far more elsewhere.
Fermi (2010) was the first design that took the compute side seriously. It added a real cache hierarchy, with an L1 per SM that could double as programmer-managed shared memory, exactly the arrangement of Part 8, and a 768 KB L2 for the whole chip. It added error-correcting memory, which no game needs and every scientific user demands. Three billion transistors, 512 lanes, 40 nm. It ran hot and late, and it also happens to be the chip on which the second accident occurred. In 2012, Alex Krizhevsky trained a neural network called AlexNet on two GeForce GTX 580s, Fermi-based gaming cards, and won the ImageNet competition by a margin that ended the argument about whether deep learning worked. Nobody at Nvidia planned for that. The lanes were there, CUDA was there, and a graduate student noticed that a convolution has the shape of a picture.
Era two: a compute chip that also does graphics (2012 to 2017)
Kepler (2012) was mostly about power. Fermi's SMs had been clocked high and ran hot, and Kepler's SMX put 192 lanes in each SM at a lower clock, which is the same bet as Part 6: more lanes, slower, cooler. The big GK110 had 7.1 billion transistors on 28 nm and could launch work from the GPU itself without going back to the CPU. Maxwell (2014) stayed on 28 nm, which forced it to find its gains in design rather than shrinkage, and the design it found is the one still in use: an SM split into four partitions, each with its own scheduler and 32 lanes, 128 per SM. Look back at the Ada SM diagram in Part 7 and you are looking at Maxwell's floor plan.
Pascal (2016) was the first generation built for the data centre first. The GP100 had 15.3 billion transistors on 16 nm and was the first Nvidia GPU with HBM, four stacks of HBM2 giving 720 GB/s, and the first with NVLink, at 160 GB/s. Both are Part 11's story beginning. Pascal also doubled the rate of FP16 arithmetic relative to FP32, the first pull on the precision lever of Part 9, because by 2016 it was clear what the customers were doing with these chips.
Era three: the AI chip (2017 onward)
Volta (2017) is the generation that defines the modern GPU, and it did so with one addition: the tensor core. The GV100 had 21.1 billion transistors on 12 nm, 80 SMs, and 640 tensor cores, each doing a 4 by 4 matrix multiply-add per tick in FP16. That single unit gave the V100 many times the matrix throughput of the P100 before it, and it is the reason "AI performance" and "graphics performance" became separate lines on a spec sheet. Volta also gave each thread its own program counter, so that diverged warps could interleave their two paths rather than finishing one before starting the other, a refinement of Part 7's divergence story. NVLink went to 300 GB/s, HBM2 to 900 GB/s.
Turing (2018) brought two of Volta's ideas to gaming cards and added a third. The GeForce RTX 20 series was the first consumer hardware with tensor cores, and the first with RT cores, units that accelerate the ray-triangle intersection tests at the heart of ray tracing. The tensor cores were put to work on DLSS, which renders a game at low resolution and uses a neural network to upscale it, and that was the first time most people ran a neural network without knowing it. 18.6 billion transistors, 12 nm.
Ampere (2020) is the chip that trained the first generation of large language models. The GA100, sold as the A100, had 54.2 billion transistors on 7 nm, an 826 mm² die close to the reticle limit of Part 2, and third-generation tensor cores that added the TF32 and BF16 formats and the 2:4 sparsity trick. Its L2 jumped to 40 MB, nearly seven times Volta's. NVLink reached 600 GB/s and HBM2 up to 2 TB/s. It could also be partitioned into up to seven separate GPUs for sharing among users, a feature called MIG. The gaming sibling, GA102 in the RTX 3090, was built on a different process at Samsung and had 10,496 lanes. If you ran a model in 2021 or 2022, it very probably ran on Ampere.
The whole run in one table
| Architecture | Year | Flagship | Transistors | Process | What it added |
|---|---|---|---|---|---|
| Tesla (G80) | 2006 | GeForce 8800 GTX | 0.68 billion | 90 nm | unified lanes, warps of 32, CUDA |
| GT200 | 2008 | GeForce GTX 280 | 1.4 billion | 65 nm | 64-bit floating point |
| Fermi | 2010 | GF100 | 3.0 billion | 40 nm | L1/shared memory, L2, ECC |
| Kepler | 2012 | GK110 | 7.1 billion | 28 nm | 192-lane SMs, GPU-launched work |
| Maxwell | 2014 | GM204 | 5.2 billion | 28 nm | the four-partition SM |
| Pascal | 2016 | GP100 | 15.3 billion | 16 nm | HBM2, NVLink, fast FP16 |
| Volta | 2017 | GV100 | 21.1 billion | 12 nm | tensor cores |
| Turing | 2018 | TU102 | 18.6 billion | 12 nm | RT cores, tensor cores for gamers, DLSS |
| Ampere | 2020 | GA100 | 54.2 billion | 7 nm | TF32, BF16, sparsity, MIG, 40 MB L2 |
What the history is telling you
Three things stand out when you line the generations up. The first is how early the important decisions were made. The warp of 32, the general-purpose lane, and CUDA all date from 2006 and 2007, and everything since has been built on them. Nvidia's advantage is not that it saw AI coming. It is that it made a general machine when a special one would have been cheaper, and then kept the software stable for two decades while the hardware underneath changed completely.
The second is that the turning points were additions, not replacements. Fermi added caches. Pascal added HBM and NVLink. Volta added the tensor core. Each time, the SM stayed recognisable, the programming model stayed the same, and old code kept running faster. That continuity is the moat of Part 10 seen from the hardware side.
The third is the shape of the curve above. From G80 to Ampere the transistor count rose about eighty times in fourteen years, mostly through shrinking. From Ampere to Blackwell it rose four times in four years, and the last step came not from shrinking but from a second die. The easy growth is over, which is exactly what Part 2 said would happen when the reticle limit was reached, and the next four parts are, among other things, the story of what Nvidia did about it.
Next: Ada Lovelace: the RTX 40 series, 76 billion transistors on a gaming chip, a cache sixteen times bigger than its predecessor's, and the moment a graphics card's most interesting new parts were the AI ones.