Home / AI Arena / Inside the GPU
Inside the GPU
A graphics card is two kilograms of metal wrapped around a square of silicon holding 76 billion switches, and every one of them can only be on or off. This series builds a GPU up from that switch, in the spirit of Jon Stokes's Inside the Machine, starting one step earlier than he did: with the relay and the vacuum tube, what they were good at, why they could never be numerous enough, and how the transistor fixed that. Then logic gates and adders, the fetch-decode-execute loop, the memory wall, why a CPU and a GPU make opposite bets, the streaming multiprocessor, the memory hierarchy, tensor cores, CUDA, and NVLink. Then the real chips: a short history from the G80 to Ampere, and a part each on Ada Lovelace, Hopper, Blackwell, and Vera Rubin. One calculation, the neuron's w × x + b from ML Basics, travels with you the whole way. Most parts have an interactive demo.
This is the hardware layer under the ML Basics, LLM Basics, and How LLMs Talk series. It can be read on its own, and links back to those where the ideas meet.
-
Part 1
Before the Transistor
There is almost certainly a vacuum tube in your kitchen and a relay in your car. Where the transistor's ancestors came from, what they were good at, what made them impossible to build a computer out of, and exactly which of their problems the transistor solved.
-
Part 2
The Switch
A graphics card is a heavy slab of metal and plastic with 76 billion switches hidden inside it, each smaller than a virus. What a transistor actually is, why computers count in twos, and what "4 nanometres" does and does not mean.
-
Part 3
From Switches to Sums
A switch is on or off. How does that become 3 + 5 = 8? Counting in twos, the handful of gates every chip is built from, a working adder you can poke at, and why a multiplier with fewer bits is the cheapest trick in AI hardware.
-
Part 4
A Machine That Follows Instructions
An adder adds whatever is on its wires, instantly and forever. A computer has to do things in order. Registers, the clock, the fetch-decode-execute loop, and a six-instruction program you can single-step through.
-
Part 5
The Memory Wall
A processor can add two numbers in a third of a nanosecond and then wait a hundred nanoseconds for the next pair to arrive. Why memory is the real bottleneck, the ladder of caches built to hide it, and the two opposite strategies that CPUs and GPUs use to cope.
-
Part 6
Two Ways to Be Fast
A desktop CPU has 16 cores. The graphics card next to it has 16,384. That is not the same word. A race between the two on the same job, where each one wins, and why a neural network is the GPU's home ground.
-
Part 7
Inside a Streaming Multiprocessor
A GPU die looks like a city grid of identical blocks. Each block is a streaming multiprocessor, the real unit of the machine. Its four schedulers, its warps of 32 threads, the trick that makes waiting free, and what happens when lockstep threads disagree about an if.
-
Part 8
Feeding the Cores
An RTX 4090 can do 41 trillion multiply-adds a second and its memory can deliver 250 billion numbers a second. Those two figures do not match. The GPU memory hierarchy, the roofline that tells you which one you are stuck on, and why a chatbot typing its answer is almost always stuck on the second.
-
Part 9
Tensor Cores
The same RTX 4090 that does 83 trillion operations a second on its ordinary lanes does 660 trillion on its tensor cores. Same silicon, same watts. How a unit that multiplies whole small matrices in one instruction works, why fewer bits per number multiplies its speed, and the ladder from FP32 to FP4 that defines each Nvidia generation.
-
Part 10
Talking to the GPU
A GPU has no operating system and cannot start itself. Everything it does begins as an instruction from the CPU next to it. Host and device, kernels, grids and blocks, a dozen lines of CUDA that map exactly onto the hardware, and why the software is the real moat.
-
Part 11
Many GPUs as One
A 405-billion-parameter model is 810 GB of weights. The biggest GPU memory is 288 GB. So the model is cut into pieces, and the pieces have to talk, fast. NVLink, NVSwitch, the superchip, the 72-GPU rack that Nvidia sells as one machine, and the ladder of bandwidths that shapes it all.
-
Part 12
A Short History
In 2006 Nvidia shipped a graphics card that could, for the first time, run a C program. Fourteen years and eight architectures later it shipped the chip that trained the first large language models. G80 to Ampere, the three eras they belong to, and the two accidents that turned a gaming company into the centre of AI.
-
Part 13
Ada Lovelace
The RTX 40 series. 76 billion transistors on a gaming chip, a cache sixteen times bigger than its predecessor's, ray-tracing hardware that fixes the warp-divergence problem, and a neural network that invents every other frame you see. The moment a graphics card's most interesting new parts were the AI ones.
-
Part 14
Hopper
The H100. Same year and same process as Ada, nearly the same transistor count, and a completely different chip. HBM3, the Transformer Engine, and the first features that let streaming multiprocessors work together. The GPU that trained the current generation of language models.
-
Part 15
Blackwell
When a chip cannot get any bigger, make two and join them. 208 billion transistors on one package, a 10 TB/s seam between the halves, four-bit arithmetic, a rack of 72 sold as one machine, and the RTX 50 series that brought the same generation to gamers.
-
Part 16
Vera Rubin
The 2026 generation, written while it is still arriving. HBM4 at 22 terabytes a second, NVLink 6, a CPU designed for the rack rather than the desktop, and a GPU with no HBM at all that exists only to read prompts. What is confirmed, what is announced, and the three trend lines that point past it.
-
Part 17
The Whole Picture
From one switch to a 72-GPU rack in a single diagram, a line-by-line guide to reading any GPU spec sheet, the three questions that tell you whether a chip suits a job, and a glossary to keep.