Home / AI Arena / Inside the GPU
Inside the GPU
A graphics card is two kilograms of metal wrapped around a square of silicon holding 76 billion switches, and every one of them can only be on or off. This series builds a GPU up from that switch, in the spirit of Jon Stokes's Inside the Machine, starting one step earlier than he did: with the relay and the vacuum tube, what they were good at, why they could never be numerous enough, and how the transistor fixed that. Then logic gates and adders, the fetch-decode-execute loop, the memory wall, why a CPU and a GPU make opposite bets, the streaming multiprocessor with AMD's compute unit and Intel's Xe-core beside it, the memory hierarchy, tensor cores, CUDA, and NVLink. Then the real chips: three short histories, Nvidia's from the G80 to Ampere, ATI's and AMD's from the Radeon 9700 to the first exascale supercomputer, and Intel's from the i740 to Arc, how a chip is manufactured and what a frame is, and a part each on Ada Lovelace, Hopper, Blackwell and Vera Rubin, each followed by AMD's answer to it: RDNA, the MI300, the MI350, and the MI400 with its Helios rack, and by Intel's alongside two of them, the Arc cards and the Ponte Vecchio, Gaudi and Crescent Island accelerators. Then two parts that step back: where the watts go, from a flipping switch to a gigawatt campus, and the NPUs and TPUs that do the same arithmetic on different terms. One calculation, the neuron's w × x + b from ML Basics, travels with you the whole way. Most parts have an interactive demo.
This is the hardware layer under the ML Basics, LLM Basics, and How LLMs Talk series. It can be read on its own, and links back to those where the ideas meet.
-
Part 1
Switches and Relays
What a computer is when you strip it to the bone, why it has to be made of switches that electricity can flip, and the first such switch, the relay, born in the telegraph and still in your car. What it is good at, what it is bad at, and why nobody could build a fast computer out of it.
-
Part 2
The Vacuum Tube
The device that ran every radio, television, and computer until the late 1950s was found by accident inside a light bulb. From Edison's lamp to the diode and the triode, what a one-way valve is for, the machines the tube built, why every one of them was a hot glass bottle with a short life, and exactly which of those problems the transistor solved.
-
Part 3
The Switch
What a transistor actually is, why computers count in twos, how a chip full of them is printed by light, what "4 nanometres" really means, why a chip cannot simply be made bigger, and why chips stopped getting faster around 2005.
-
Part 4
From Switches to Sums
A switch is on or off. How does that become 3 + 5 = 8? Counting in twos, the handful of gates every chip is built from, a working adder you can poke at, and why a multiplier with fewer bits is the cheapest trick in AI hardware.
-
Part 5
A Machine That Follows Instructions
An adder adds whatever is on its wires, instantly and forever. A computer has to do things in order. Memory cells built from two NOT gates, the clock and the flip-flop, the carry problem and its cure, the program counter and the decoder built from gates, and a toy processor you can single-step through, loop and all.
-
Part 6
What Makes a GPU a GPU
A CPU and a GPU run the same fetch-decode-execute loop from Part 5. The difference starts with the shape of the work, the same routine over millions of independent items, and follows from it, how many execution units one front end feeds, what surrounds the loop, and what the memory looks like. One instruction across 32 lanes, DDR against GDDR against HBM, and what the number on a RAM stick really means.
-
Part 7
The Memory Wall
A processor can add two numbers in a third of a nanosecond and then wait a hundred nanoseconds for the next pair to arrive. Why memory is the real bottleneck, the ladder of caches built to hide it, and the two opposite strategies that CPUs and GPUs use to cope.
-
Part 8
Two Ways to Be Fast
A desktop CPU has 16 cores. The graphics card next to it has 16,384. That is not the same word. A race between the two on the same job, where each one wins, and why a neural network is the GPU's home ground.
-
Part 9
Inside a Streaming Multiprocessor
A GPU die looks like a city grid of identical blocks. Each block is a streaming multiprocessor, the real unit of the machine. Its four schedulers, its warps of 32 threads, the trick that makes waiting free, and what happens when lockstep threads disagree about an if.
-
Part 10
Inside an AMD Compute Unit
AMD builds its GPUs from compute units rather than SMs, and runs threads in wavefronts of 64 or 32 rather than warps of 32. How GCN's four SIMDs took turns, what the scalar unit is for, why RDNA moved to wave32 and paired its compute units, why the data-centre chips kept wave64, and how one block fills both a Radeon and an Instinct MI300X.
-
Part 11
Inside an Intel Xe-core
The third company's building block. Intel's Xe-core, its vector engines and matrix engines, hardware threads whose width the compiler chooses, a register file with a fixed seat for every thread, memory reached by sending messages, and how Battlemage rebuilt the core with half as many engines twice as wide.
-
Part 12
Feeding the Cores
An RTX 4090 can do 41 trillion multiply-adds a second and its memory can deliver 250 billion numbers a second. Those two figures do not match. The GPU memory hierarchy, the roofline that tells you which one you are stuck on, and why a chatbot typing its answer is almost always stuck on the second.
-
Part 13
Tensor Cores
The same RTX 4090 that does 83 trillion operations a second on its ordinary lanes does 660 trillion on its tensor cores. Same silicon, same watts. How a unit that multiplies whole small matrices in one instruction works, why fewer bits per number multiplies its speed, and the ladder from FP32 to FP4 that defines each Nvidia generation.
-
Part 14
Talking to the GPU
A GPU has no operating system and cannot start itself. Everything it does begins as an instruction from the CPU next to it. Host and device, kernels, grids and blocks, a dozen lines of CUDA that map exactly onto the hardware, and why the software is the real moat.
-
Part 15
Many GPUs as One
A 405-billion-parameter model is 810 GB of weights. The biggest GPU memory is 432 GB. So the model is cut into pieces, and the pieces have to talk, fast. NVLink, NVSwitch, the superchip, the 72-GPU rack that Nvidia sells as one machine, and the ladder of bandwidths that shapes it all.
-
Part 16
A Short History
In 2006 Nvidia shipped a graphics card that could, for the first time, run a C program. Fourteen years and eight architectures later it shipped the chip that trained the first large language models. G80 to Ampere, the three eras they belong to, and the two accidents that turned a gaming company into the centre of AI.
-
Part 17
ATI and AMD
Nvidia's one lasting rival in big GPUs began as a Canadian graphics-card maker. The chip that beat Nvidia to DirectX 9, the console GPU with unified shaders a year before the G80, the takeover that nearly sank AMD, the small-die bet, the first HBM, and the long climb from gaming rival to the fastest supercomputers and gigawatt contracts.
-
Part 18
Intel and Graphics
The company that has shipped more GPUs than anyone, and spent a quarter of a century trying to build a big one. The i740 and the retreat to integrated graphics, the GPU moved onto the processor die, Larrabee's x86 gamble and the Xeon Phi it became, the Xe architecture and the Arc cards, Aurora, Gaudi, and the retrenchment of 2025 and 2026.
-
Part 19
How a Chip Is Made
From a bucket of sand to a graphics card. Where the wafer comes from, the machine that prints it with light, what "die" means and how many fit on a wafer, why an RTX 4090 has 128 working SMs out of 144, what binning is, and how the memory ends up glued to the chip. The words every spec sheet uses, with numbers attached.
-
Part 20
The Graphics Pipeline
The G in GPU. What a frame actually is, why everything on a screen is made of triangles, the six stages that turn a list of corners into eight million coloured pixels sixty times a second, and why that job has exactly the shape that made the GPU. Plus the parts of a modern die that still do nothing but draw.
-
Part 21
Ada Lovelace
The RTX 40 series. 76 billion transistors on a gaming chip, a cache sixteen times bigger than its predecessor's, ray-tracing hardware that fixes the warp-divergence problem, and a neural network that invents every other frame you see. The moment a graphics card's most interesting new parts were the AI ones.
-
Part 22
Radeon and RDNA
AMD's gaming chips from RDNA 2 to RDNA 4, set beside Ada. A cache that lets a 256-bit bus do a 512-bit bus's job, ray tracing done half in hardware, the first gaming GPU built from chiplets and the retreat to one die, FP8 and machine-learning upscaling on a Radeon, the same design in every current console, and what all of it means for running a model at home.
-
Part 23
Intel Arc
Intel's graphics cards, set beside Ada and RDNA. Alchemist's large die and troubled drivers, ray tracing done fully in hardware with a unit that sorts rays before shading them, Battlemage doing more with less, the first AV1 encoder, Arc in laptops from Meteor Lake to Panther Lake, and 32 GB cards for running models at home.
-
Part 24
Hopper
The H100. Same year and same process as Ada, nearly the same transistor count, and a completely different chip. HBM3, the Transformer Engine, and the first features that let streaming multiprocessors work together. The GPU that trained the current generation of language models.
-
Part 25
Instinct MI300
AMD's answer to Hopper, and its first real success in AI. Where it came from, in the chip that built the first exascale supercomputer. One package built two ways, as a GPU and as a CPU and GPU sharing one memory. 192 GB when the H100 had 80, and what that let people do. What held it back, and the MI325X that followed.
-
Part 26
Intel in the Data Centre
Intel's answers to Hopper. Ponte Vecchio, a GPU of forty-seven tiles on five processes, and Aurora, the exascale machine built from it and what its power bill says. Gaudi, an AI accelerator that was not a GPU at all. The cancellations, and Crescent Island, a 2026 inference card that trades HBM for 160 GB of laptop memory.
-
Part 27
Blackwell
When a chip cannot get any bigger, make two and join them. 208 billion transistors on one package, a 10 TB/s seam between the halves, four-bit arithmetic, a rack of 72 sold as one machine, and the RTX 50 series that brought the same generation to gamers.
-
Part 28
Instinct MI350
AMD's answer to Blackwell. CDNA 4 on 3 nm, eight compute dies on two base dies, 288 GB of HBM3E, a compute unit that doubles its matrix work and halves its 64-bit work, six-bit numbers at four-bit speed, 1,400 watts, and how close it came to Nvidia on paper and on the benchmark.
-
Part 29
Vera Rubin
The 2026 generation, written while it is still arriving. HBM4 at 22 terabytes a second, NVLink 6, a CPU designed for the rack rather than the desktop, and a GPU with no HBM at all that exists only to read prompts. What is confirmed, what is announced, and the three trend lines that point past it.
-
Part 30
Instinct MI400 and Helios
AMD's 2026 generation, written as it ships. A compute die that finally works like a Radeon's, 432 GB of HBM4 on twelve stacks, a 72-GPU rack joined through Ethernet switches carrying a new protocol, Nvidia's numbers beside AMD's, and the gigawatt contracts that were signed before the hardware existed.
-
Part 31
Power
Where the 450 watts on a graphics card actually go, why moving a number costs more energy than multiplying it, what TDP and SXM mean, and the ladder from a 700 W chip to a 10 kW server, a 120 kW rack that needs liquid to stay cool, and campuses now approaching a gigawatt. Where the water goes, and what one chatbot answer costs in electricity and in water.
-
Part 32
NPUs, TPUs and the Chips in Your Phone
Every phone and most new laptops have a neural processing unit next to the GPU. What an NPU is, why a five-watt chip can advertise 50 TOPS, how a systolic array does w × x + b with no warps and no caches, what TOPS actually measures, where the NPU cannot follow the GPU, and how Google's TPU is the same idea grown to a rack.
-
Part 33
The Whole Picture
From one switch to a 72-GPU rack in a single diagram, a line-by-line guide to reading any GPU spec sheet, the three questions that tell you whether a chip suits a job, and a glossary to keep.