Part 1 left a strange object on the desk: a huge file of frozen numbers that contains everything a language model knows and cannot do anything with it. A recipe book on a shelf. The obvious next question: what actually makes the numbers compute?
The engine
The answer is a second program, and it deserves a name most people never hear: the inference engine. "Inference" is the field's word for using a trained model, as opposed to training it. The engine's job is to load the weights into memory and do the arithmetic of prediction. Feed it text, and it pushes that text through the billions of frozen numbers, multiply and add, layer after layer, until out comes the ranked list of next tokens from Part 1. Pick a token, append it, run the whole thing again for the token after that. The engine does this and only this, at ferocious speed.
One engine is worth knowing by name: llama.cpp, an open-source project that became famous for letting ordinary home computers run serious models. You do not need a longer list. What matters is the division of labour, because it is the same division you will see at every layer of this series: the file provides the knowledge, the engine provides the muscle, and neither is any use alone.
This is the kitchen in our running analogy. The recipe book sits inert until a kitchen cooks from it. And the analogy breaks in exactly one place, worth saying out loud: a real kitchen improvises, tastes, and adjusts, while the engine does precisely the arithmetic the weights dictate and nothing else. There is no chef's judgement in here. The judgement, such as it is, is already baked into the numbers.
Why a graphics card, of all things
You may have noticed that local AI enthusiasts talk about graphics cards constantly, and there is a one-sentence reason. Producing a single token takes billions of small multiplications, and a graphics card, built to colour millions of pixels at once, happens to be a machine for doing thousands of simple calculations at the same time. The shape of the work fits. A regular processor can run a model too, only slower, the way one very good cook can serve a banquet if you give them all night.
The other half of the story is memory. The whole recipe book has to sit somewhere the engine can reach quickly, which is why the first question of running models at home is never "is my computer clever enough" but "does my graphics card have enough memory to hold the file". That question leads directly to the strangest thing on any model download page.
The same brain, stored with fewer decimal places
Go to download a model and you will not find one file. You will find a wall of them: the same model name followed by cryptic suffixes like Q8, Q6, Q4. These are quantised versions: the same weights, stored with fewer decimal places. Think of rounding 3.14159265 down to 3.14 four billion times. The knowledge survives almost intact, but the file shrinks dramatically, and so does the memory needed to run it.
Does the rounding hurt? Slightly, and it would be dishonest to say otherwise. A heavily compressed model is numerically a little different from the original, and at the extreme end (Q2) the rough edges start to show. But the middle grades are, in casual use, usually indistinguishable from the full file, which is why Q4 became the default of the local AI world. Same brain, cheaper shelf. There is no need to go deeper than that: when you see the wall of suffixes, you now know it is one model at five levels of rounding, not five models.
What the kitchen still cannot do
Here is where the layers get interesting. Suppose you have the file and the engine, and you feed our running example straight in, raw: What's the weather in Paris?
What comes back is not an answer. The engine dutifully continues the text, and a plausible continuation of a lone question on a page is more text like it. You might get a list of further questions about Paris, or a paragraph that begins "This is a common question among travellers...". The machinery worked perfectly. Nothing in it knows a conversation is happening, that there are two speakers, or that one of them is supposed to be helpful. A machine that finishes text has been handed text, and it finished it.
Turning that into the experience of talking to someone, the turn-taking, the remembered name, the personality, is not done by the model or the engine at all. It is done with an idea so low-tech it feels like a prank, and it is the subject of the next part.
Next: The Illusion of Conversation: the model has no memory of you, none at all, and what your chat app does about that every single time you press send.