Here is a puzzle. In December 2024 NVIDIA ran Llama 3.3 70B, a 70-billion-parameter model, on a single H200 graphics card and measured about 51 tokens a second. Then they loaded one extra file next to it, a 1-billion-parameter cousin from the same family, and ran the same model on the same card again. 182 tokens a second. Three and a half times faster, with the same weights at the same precision, and every token in the answer still chosen by the 70B model. Nothing was pruned and no quality was traded away.

The technique is called speculative sampling (you will also see speculative decoding, and Andrej Karpathy calls it speculative execution, all the same idea). It sounds like it should not work, then turns out to rest on a single unintuitive fact about how graphics cards run language models. This article is my attempt to explain it from scratch, with worked examples, including the bad round when every guess is wrong and the whole batch goes in the bin. At the end I go through who actually uses it, because the answer turns out to be nearly everyone.

Why a model writes one token at a time

Start with what a language model does when it answers you. Part 1 of How LLMs Talk covered the basic move: given everything written so far, the model produces a ranked list of what the next token should be, one is picked, it gets appended to the text, and the whole thing runs again for the token after that. A token is a word or a piece of a word. For the pictures in this article I will treat each word as one token, which is a simplification but not a misleading one.

one forward pass: text in, ranked list of next tokens out text so far The quick brown the model layer after layer of multiply and add all the weights, in GPU memory read once per pass ranked list of next tokens fox 62% dog 18% cat 9% horse 4%, and so on pick one, append it, run the whole thing again The quick brown fox one pass per token, every pass reads every weight
One forward pass. The text so far goes in, every weight is read from memory on the way through, and out comes a ranked list of candidates for the next token. Pick one, append it, and do the whole thing again for the token after that.

Each of those steps is a complete run of the network, what engineers call a forward pass. Every one of the model's billions of weights is read from memory and used once. For a 70B model stored at 16 bits per weight that is 140 GB of numbers read from memory for every single token. Write a 300-token answer, read 140 GB three hundred times.

It is worth being precise about who is reading what, because the rest of this article turns on it. The reader is the graphics card's processing cores, the thousands of small arithmetic units that do the multiplying and adding. The memory is the graphics card's own memory, the VRAM (video RAM) soldered around the chip, not the computer's main RAM. On a desktop card that is a ring of memory chips around the GPU, and on a data-centre card like the H200 it is stacks of HBM (High Bandwidth Memory) bonded right next to it. The weights are copied into VRAM once, when the model loads, and stay there for the whole conversation, which is why "does my card have enough memory to hold the file" is the first question of running a model at home (How LLMs Talk, Part 2). Main RAM only gets involved when the model does not fit on the card, and then everything slows down badly, because the weights have to come across the slot that connects the card to the rest of the computer, which is many times slower again.

Each pass, the cores pull every weight across the wires from VRAM onto the chip, use it in one multiply-add, and let it go. The chip cannot keep the weights: its on-chip caches hold tens of megabytes, and the model is 140 gigabytes, so there is no option but to fetch the whole lot again for the next token. The 300 reads are not a bookkeeping quirk. They are 300 separate trips for the same 140 GB.

one pass, seen from the hardware: every weight makes the same trip VRAM the card's own memory 140 GB of weights the memory bus 4.8 TB/s on an H200 140 GB per pass, about 29 ms the GPU chip on-chip cache tens of MB too small to keep them weight in, one multiply-add, weight dropped thousands of cores, mostly waiting for the next weight to arrive next token: the same 140 GB makes the same trip again the trip, not the arithmetic, is what sets the speed
The trip every weight makes on every pass. VRAM holds all 140 GB, the bus carries them to the chip at a fixed rate, and the cores use each one in a single multiply-add and let it go, because the on-chip cache is thousands of times too small to keep them. Next token, same trip.

And the reason that is bad is simply that it takes time, with a floor you cannot engineer around. A memory system moves a fixed number of bytes per second, its bandwidth. The H200 from the opening puzzle moves about 4.8 TB/s, which is a staggering figure and still means 140 GB takes about 29 milliseconds to read. So that card cannot run a 70B model at 16 bits faster than about 34 passes a second, which is 34 tokens a second, however fast its arithmetic is. A 300-token answer is 300 of those reads back to back, around nine seconds, and nearly all of it spent waiting for bytes rather than computing.

That is what engineers mean when they say generation is memory-bound. The arithmetic for one token, billions of multiply-adds, is work the cores could finish in a small fraction of the time it takes to fetch the weights, so the cores spend most of every pass idle. Part 12 of Inside the GPU works through the same sum for a home setup: an 8-billion-parameter model at 16 bits is 16 GB, an RTX 4090 reads its memory at about 1 TB/s, so 60 tokens a second is that card's ceiling no matter how fast the arithmetic is. In the language of Part 7, the model is stuck behind the memory wall.

So the cost of producing a token is dominated by reading the weights, and reading the weights has nothing to do with how many tokens you push through. That leads to the fact everything else hangs on.

The unintuitive fact

If you give the model one token and ask it to run a forward pass, it reads all the weights once. If you give it six tokens and ask it to run one forward pass over all six at the same time, it still reads all the weights once. The arithmetic goes up six-fold, but the arithmetic was never the bottleneck. The six-token pass takes very nearly the same time as the one-token pass.

what one forward pass costs (not to scale, but not far off) 1 token read every weight from memory 6 tokens, one pass read every weight from memory maths time arithmetic
The bill for a forward pass is almost all memory reads. Pushing six tokens through instead of one adds a sliver of arithmetic and nothing else.

You have already met this fact twice without it being named. It is why prefill, the phase where the model reads your whole prompt before writing anything, takes one pass for a thousand-token prompt rather than a thousand. And it is why servers batch many users together, reading the weights once and using them for sixty-four people's tokens, which is the reason a cloud API is so much cheaper per token than running the model yourself. Karpathy's post from August 2023, the best short explanation of this topic I know, rests the whole idea on this one observation: running the model on one token takes about as long as running it on a batch of several, so if you are going to pay for reading the weights you might as well use them on a batch.

The catch is obvious once you see it. You cannot just hand the model six empty slots and ask for the next six tokens in one pass, because token number two depends on token number one, which has not been chosen yet. Generation is sequential by nature. Each token needs to know what came before it.

Unless you guess.

The idea: guess with a small model, check with the big one

Load a second, much smaller model alongside the big one. Same family, same vocabulary, a fraction of the size. Call the big one the target model (it is the one whose answers you actually want) and the small one the draft model. Then each round of generation goes like this.

First, the draft model writes ahead on its own, one token at a time, the normal slow way, for a few tokens. Call that number K. Something like five is typical, and engines let you tune it. The draft model is small, so each of its passes reads a small file and takes a fraction of the time of a big-model pass.

Second, feed the original text plus all K draft tokens into the target model as one batch, one forward pass. Because of the unintuitive fact, this costs about the same as generating a single token the ordinary way.

Now the detail that makes the whole thing work, and that ordinary chat hides from you. A transformer does not produce one prediction at the end of its input. It produces a prediction at every position, in the same pass: after token one, a ranked list for token two, after token two, a ranked list for token three, and so on down the line. In ordinary generation only the last of those is used, because the earlier positions are text you already have, so the rest are computed and thrown away. (It is also how the model was trained, scored on every position of every training text at once, which is why the ability is there at all.) Speculation puts the discarded predictions to work. Feed in "The quick brown fox" plus the draft "jumps over the lazy dog", and the one pass hands back the target model's own ranked list after "fox", after "fox jumps", after "fox jumps over", and so on, one for every draft token and one more at the end, all computed together.

one pass over the batch: a prediction at every position, not just the last input to the target model: the text so far, then the five draft tokens The quick brown fox jumps over the lazy dog what the model says comes next, at every position, all from the same pass predictions here too, but this text is already known, so ignored jumps 71% is 9% ran 6% over 80% on 6% high 4% the 85% a 7% two 2% lazy 55% fence 12% dog 9% dog 90% cat 4% fox 2% . 60% and 15% again 8% compare each list with the draft token in the next column along draft says jumps ✓ draft says over ✓ draft says the ✓ draft says lazy ✓ draft says dog ✓ no draft left bonus: . six ranked lists from one read of the weights, where ordinary generation keeps only the last one
Step by step through one verification pass. Every column gets its own ranked list, made from everything to its left. The list under "fox" is checked against the draft's "jumps", the list under "jumps" against the draft's "over", and so on. The list under the last draft token has nothing to check against, so its top entry becomes a free extra token. The percentages are illustrative.

Third, compare, left to right. At each position, does the target model's own choice match the draft token sitting there? If yes, keep it and move on. At the first mismatch, stop. Throw away that draft token and everything after it, and in its place use the target model's own choice at that position, which you already have for free from the same pass.

The result is that one big-model pass yields somewhere between one token and K plus one tokens, and the text that comes out is text the target model would have written itself. Let me walk through three rounds, a good one, a mixed one, and the worst case.

A good round: six tokens for the price of one

The text so far is "The quick brown fox". The draft model proposes five tokens: jumps, over, the, lazy, dog. All five go into the target model with the original text in one pass, and the ranked lists come back as in the figure above. Keep only the top entry of each list and the comparison looks like this.

a good round: draft length 5, everything accepted goes into the target model The quick brown fox jumps over the lazy dog five draft tokens, written by the small model target model's own choice here jumps over the lazy dog . ✓ ✓ ✓ ✓ ✓ bonus result: 6 tokens appended, for one pass through the big model
Each prediction in the middle row was made after reading everything to its left in the top row, and sits under the draft token it is compared with. When all five agree, the prediction at the far end is a sixth token for free.

The target model's own choice after "fox" is "jumps". Match. After "fox jumps" it is "over". Match. "the", "lazy", "dog", all matches. And because the pass also produced a prediction at the very last position, after "dog", we get a sixth token, a full stop, for nothing. Six tokens appended to the text for one read of the big model's weights, plus five small reads for the draft.

Nothing about this cuts a corner on quality. Every one of those six tokens is a token the target model itself chose at that position. The draft model only proposed them. The target model verified them.

A mixed round: keep two, correct one, bin two

Now the text is "The weather in Paris today is". The draft model proposes sunny, and, warm, with, a. Into the target model in one pass.

a mixed round: two accepted, one corrected, two thrown away goes into the target model The weather in Paris today is sunny and warm with a target model's own choice here sunny and mild ignored ignored ✓ ✓ ✗ use "mild" instead thrown away result: 3 tokens appended ("sunny and mild"), for one pass through the big model
The first disagreement ends the round. The target model's own choice at that position replaces the draft token, and everything to the right of it is discarded unread.

After "is", the target model also says "sunny". After "sunny" it also says "and". But after "sunny and" the target model's own choice is "mild", not "warm". That is the first disagreement, so the round ends there. We keep "sunny" and "and", we take the target model's "mild" as the third token, and the draft's "with" and "a" go in the bin. They were predictions built on top of "warm", which the target model never wrote, so they are meaningless now even if they happened to be sensible words.

Three tokens for one big-model pass. Still a win over the ordinary one token per pass, just a smaller one.

The worst round: everything in the bin

Suppose the conversation so far has established that the user is a Rust developer, and the text now ends "My favourite programming language is". The small draft model, with a weaker grasp of the earlier context, proposes Python, because, it, is, easy.

the worst round: the very first draft token is rejected goes into the target model My favourite programming language is Python because it is easy target model's own choice here Rust ignored ignored ignored ignored ✗ use "Rust" instead all four thrown away result: 1 token appended ("Rust"), five draft passes and most of a big pass wasted
Zero draft tokens survive. The round still produces one token, the target model's own, so the text never goes backwards. But everything spent on the draft was for nothing.

The target model's own choice after "is" is "Rust". The first draft token is wrong, so all five are discarded. The round produces exactly one token, "Rust", which is what ordinary generation would have produced in one pass anyway. But we paid for five draft passes to get it, plus a slightly more expensive big pass (the arithmetic for six positions rather than one), plus some housekeeping. The engine also has to clean up: during the pass the target model wrote internal notes (the KV cache, its memory of each position) for all five draft positions, and the notes for the rejected ones have to be thrown out so they do not pollute the next round.

Two things are worth noticing about the worst case. The output is still correct. Speculation can never make the text worse, because the target model has the last word at every position, and it can never produce zero tokens, because the correction is always there. What it can do is make generation slower. If the draft model is wrong most of the time, you are doing all the ordinary work plus the drafting on top, and you would have been better off without it. Georgi Gerganov found exactly this when he built the first implementation in llama.cpp at the end of August 2023: on code, where the next token is often obvious, the draft was accepted more than 75% of the time and a 34B model ran about twice as fast with a 7B draft. On free-form prose, the acceptance rate dropped and the method stopped paying for itself.

When the big model is not sure either

Everything above used the simplest comparison rule: does the target model's top choice equal the draft token? That is the right rule when the model is set to always pick its most likely token, which engineers call greedy decoding. Most chat systems do not do that. They sample, picking from the ranked list with a bit of randomness controlled by the temperature setting, which is what makes the same question produce slightly different answers each time. Part 13 of ML Basics covers where those probabilities come from.

With sampling, "did it match" is the wrong question, because there is no single right answer at a position, only a probability for each candidate. Take the Paris example again. Suppose the draft proposes "sunny", and the target model's own list after "today is" puts "mild" first at 40% and "sunny" second at 30%. Under the greedy rule that is a mismatch, and "sunny" goes in the bin. But the target model, writing on its own with sampling on, would have said "sunny" three times in ten. If we bin it every time the draft proposes it, the output drifts away from the target model's: it becomes a model that never writes "sunny" here, when the real one often would. Keeping it every time is wrong in the other direction. The draft proposed "sunny" because it rates it at 60%, twice as often as the target does, so always keeping it makes "sunny" turn up far too often. The right answer is in between: keep it some of the time, in exactly the proportion that makes the mix of words come out as the target model's own.

after "The weather in Paris today is", the draft proposed "sunny" draft model's list sunny 60% mild 15% snowy 15% grey 5% cold 5% "sunny" is top of this list and second on this one target model's list mild 40% sunny 30% grey 20% cold 8% snowy 2% throw it away every time? the output never says "sunny" here, though the target would, 3 times in 10 keep it every time? "sunny" turns up twice as often as the target would say it keep it half the time (30 ÷ 60): the mix comes out exactly as the target's own
A draft token that is second on the target's list is neither right nor wrong. Always binning it or always keeping it both distort the output. Keeping it in proportion to how the two models rate it is what preserves the target model's mix of words.

The 2022 Google paper and the 2023 DeepMind paper (more on both below) worked out what that proportion has to be, and the rule is simple to state. For each draft token, look at how likely the target model thought it was, and how likely the draft model thought it was.

If the target model rates the token at least as likely as the draft did, accept it. If the target model rates it less likely, accept it with a probability equal to the ratio, target over draft. A token the draft gave 60% and the target gives 30% is kept half the time. A token the draft gave 15% and the target gives only 2% is kept about one time in eight, so a draft token the target thinks is a poor fit is nearly always binned, without ever being banned outright. When a token is rejected, the replacement is not simply the target model's top pick. It is drawn from the words the target model liked more than the draft did, the ones the draft under-proposed.

Candidate after "today is" Draft model Target model Ratio, target over draft Kept if drafted?
sunny 60% 30% 0.5 half the time
mild 15% 40% 2.7 always (anything above 1 means always)
grey 5% 20% 4.0 always
cold 5% 8% 1.6 always
snowy 15% 2% 0.13 about 1 time in 8, nearly always binned

Two rows are worth a second look. "mild" is the target model's favourite and the draft rates it low, so on the occasions the draft does propose it, it sails through. "snowy" is the opposite: the small model has a weakness for snow in Paris, the target thinks it almost never fits, and the ratio of 0.13 means that roughly seven proposals in eight are thrown out. Not eight in eight, because the target does give it 2%, and the rule is careful to preserve even that.

In this example the draft proposes "sunny" and it gets rejected on the coin flip. The replacement is drawn from where the target model exceeded the draft: "mild" by 25 points, "grey" by 15 and "cold" by 3, so "mild" a little over half the time, "grey" about a third, "cold" now and then, never "sunny" again and never "snowy". It takes a short proof to show it, but the upshot is that over many runs the mix of tokens that comes out of this procedure is exactly the mix the target model would have produced sampling on its own. The randomness is preserved, not just the top answer. The vLLM project, which runs this in production, states it as "theoretically lossless up to the precision limits of hardware numerics" in its documentation, and the precision caveat is there because floating-point arithmetic on a graphics card is not perfectly repeatable in any case.

This is the detail that earned the technique its name. "Speculative decoding" was Google's term and covers the greedy case cleanly. "Speculative sampling" was DeepMind's, and it emphasises that the random version works too. Karpathy borrowed "speculative execution" from processors, which guess which branch a program will take and run ahead on the guess, throwing the work away if they were wrong. Same shape of idea, different machine.

How much faster, and when it goes wrong

Everything comes down to the acceptance rate, the fraction of draft tokens the target model agrees with. The maths for the expected number of tokens per round is in the Google paper and it is not frightening, but a table makes the point better than the formula. Assume a draft of five tokens, a draft pass costing a tenth of a target pass, and the verification pass costing about the same as an ordinary single-token pass. Those are rough but realistic numbers for a 7B draft on a 70B target.

Acceptance rate Tokens per round (average) Cost per round, in target passes Speed relative to no speculation
90% 4.7 1.5 about 3.1x
80% 3.7 1.5 about 2.5x
60% 2.4 1.5 about 1.6x
40% 1.7 1.5 about 1.1x
20% 1.2 1.5 about 0.8x, slower
0% 1.0 1.5 0.67x, a third slower

Two lessons fall out. First, the speedup is very sensitive to acceptance rate in the range that real systems live in, which is why so much engineering goes into making the draft agree with the target. Second, there is a crossover below which speculation hurts, and in this toy model it sits somewhere around a 35% acceptance rate. That is not an exotic situation. Creative writing at high temperature, where even the big model is unsure what comes next, is exactly where a small draft model struggles, and it was the case that disappointed in the llama.cpp experiments. LM Studio's documentation says the same thing in one line: if too many draft tokens get rejected, total generation speed can drop.

What raises the acceptance rate is making the draft think like the target. The simplest lever is choosing the draft from the same model family, trained on the same data, so it has the same habits. Perplexity went a step further for its Sonar models and fine-tuned a 1B draft on the same dataset as the target. Together AI's ATLAS system goes further still and keeps training its draft on live traffic, on the reasoning that a draft which has seen what users actually ask will agree with the target more often. The other lever is the draft length K. Longer drafts give bigger wins when they are accepted and bigger waste when they are not, so engines tune it, and some adjust it on the fly based on how confident the draft is.

Two costs are easy to forget. The draft model takes memory. A 7B draft next to a 34B target adds something like 20% to the memory needed, which was the first question asked in the Reddit thread that followed the llama.cpp release. And speculation is built for the batch size one case, a single conversation on a single machine, which is most local use and much of latency-sensitive cloud use. A busy server already batching sixty-four users has filled the idle arithmetic with other people's tokens, so there is less spare capacity for speculation to borrow. The vLLM documentation is explicit that the technique is aimed at reducing per-token latency under medium-to-low traffic.

Drafts without a draft model

Once you see that the target model only needs some source of plausible guesses, you start to notice that the guesses can come from anywhere, and much of the research since 2023 has been about finding cheaper sources than a second model.

Medusa bolts a few extra prediction heads onto the target model itself, each guessing one more token ahead, so there is no separate draft model to load. EAGLE trains a very small network to predict the next token from the target model's internal state rather than from the text, which turns out to be much more accurate than a separate model guessing from words alone. Apple's ReDrafter uses a small recurrent network as the drafter and searches several candidate continuations at once. Prompt lookup, also called n-gram speculation, has no model at all: it guesses that the next few tokens will repeat a phrase that already appeared in the prompt, which is a surprisingly good bet when you are asking a model to edit a document or rewrite code, since most of the output is copied from the input. DeepSeek trained the ability in from the start: DeepSeek-V3 has a multi-token prediction module that predicts the token after next during training to make the model better, and at inference it is repurposed as a built-in drafter. The technical report says the second token is accepted 85% to 90% of the time, which the team says delivers 1.8 times the tokens per second.

The most unusual drafter is you. OpenAI's Predicted Outputs feature, released in November 2024, lets a developer send along text they expect the model to mostly reproduce, such as a file they are asking it to lightly edit. The service uses that text as the draft. An OpenAI engineer confirmed on launch that they were using the prediction to do speculative decoding and validate large batches in parallel. The pricing tells you the worst case has a bill: rejected prediction tokens are charged at output rates, so a bad prediction costs more than no prediction. Simon Willison's test run came back in 3.3 seconds instead of 5.2, with 49 predicted tokens accepted and 107 rejected, and cost about 70% more. Faster and pricier, which is precisely the trade the worst-case round describes.

Who uses it

The short answer is that by 2024 speculative decoding had become a standard part of serving language models, and the long answer is a tour of the industry.

Who What they do with it Result they report Source
Google Invented it (2022 paper), runs it behind AI Overviews in Search 2x to 3x in the paper, "faster than before" in Search Google Research blog, paper
OpenAI Predicted Outputs: the developer's own text is the draft Faster when the prediction is good, costlier when it is not OpenAI docs, Simon Willison's write-up
Apple ReDrafter, built into NVIDIA's TensorRT-LLM, used on a production model 2.7x more tokens per second Apple Machine Learning Research
NVIDIA Draft-target speculation in TensorRT-LLM Llama 3.3 70B, 51 to 182 tokens/s on one H200 with a 1B draft NVIDIA technical blog
Groq A 70B endpoint labelled "specdec" About 250 to about 1,660 tokens/s from software alone Groq announcement, model page
Cerebras Part of a release that tripled Llama 3.1 70B speed 2,100 tokens/s, speculation credited alongside other work Cerebras blog
DeepSeek Multi-token prediction module reused as a drafter 85% to 90% acceptance, 1.8x tokens per second DeepSeek-V3 technical report
Perplexity Sonar models drafted by a fine-tuned 1B model Deployed in production, no figure published Perplexity research blog
Together AI Speculators that keep learning from live traffic Up to 500 tokens/s on DeepSeek-V3.1, from a baseline near 105 Together AI blog
llama.cpp, LM Studio, vLLM, Hugging Face Built-in options for anyone to switch on 1.5x to 3x is typical in their own benchmarks llama.cpp PR, LM Studio, vLLM docs, Hugging Face

The idea has a tidy history. In 2018 Mitchell Stern, Noam Shazeer and Jakob Uszkoreit at Google published blockwise parallel decoding, which predicted several tokens ahead and kept the longest prefix the model agreed with, but only for greedy decoding. In November 2022 Yaniv Leviathan, Matan Kalman and Yossi Matias at Google Research published Fast Inference from Transformers via Speculative Decoding, which added the sampling rule that keeps the output distribution intact and reported 2x to 3x on T5-XXL, an 11-billion-parameter translation and summarisation model. Three months later Charlie Chen and colleagues at DeepMind, John Jumper among them, independently published Accelerating Large Language Model Decoding with Speculative Sampling, the same idea applied to the 70B Chinchilla model, with a 2x to 2.5x speedup. Google says in a 2024 retrospective that the technique now helps AI Overviews in Search produce results faster and reduces the number of machines needed to serve them.

The moment it went mainstream outside the labs was the last day of August 2023. Gerganov posted a demo of the llama.cpp implementation running a Code Llama 34B model with a 7B draft on an Apple M2 Ultra, roughly 10 tokens a second going to roughly 20, and Karpathy wrote the explainer post that this article leans on. Hugging Face had shipped the same idea into its transformers library that May under the name assisted generation. By the end of 2024 Groq and Cerebras were posting four-digit tokens-per-second figures for 70B models with speculation doing much of the lifting, and in February 2025 LM Studio put a draft-model dropdown in a desktop app. Apple noted in its own foundation models overview that its on-device figures, 30 tokens a second on an iPhone 15 Pro, were measured before applying token speculation, which improves them further.

If you want to see it yourself, Part 5 of How LLMs Talk covered what LM Studio and the llama.cpp server are. In LM Studio, load a main model, pick a much smaller one from the same family as the draft (the documentation suggests Llama 3.2 1B drafting for Llama 3.1 8B), and turn on Visualize accepted draft tokens. In the llama.cpp server the flags are --model-draft for the second file and --draft-max and --draft-min for how far ahead to guess. Then ask for some code and watch the colours, and ask for a poem and watch them change.

Key takeaways

  • Generating a token means reading every weight once, and that read is the cost. Reading them once for six tokens costs about the same as reading them once for one.
  • Generation cannot use that fact directly because each token depends on the one before it. A small draft model supplies guesses so that the big model can check several at once.
  • The big model keeps the longest run of guesses it agrees with, corrects the first one it does not, and discards the rest. Every round yields at least one token and never changes what the big model would have said.
  • With sampling turned on, a probability-based acceptance rule keeps the randomness of the big model intact too.
  • The worst case is a rejected first guess: one token for the price of a round of drafting. If that happens too often, speculation makes things slower, which is why acceptance rate is the number everyone tunes for.
  • Drafts can come from a smaller model, extra heads on the big model, a tiny network reading its internal state, the prompt itself, or text the developer supplies. Google, OpenAI, Apple, NVIDIA, Groq, Cerebras, DeepSeek, Perplexity and the local-AI tools all use some version of it.

What the technique is really about

What I find satisfying about speculative sampling is that it is not a model technique at all. The model is unchanged, the weights are unchanged, the output is unchanged. It is a hardware trick wearing a model costume. A graphics card is a machine that is fast at doing many things at once and slow at fetching the things to do them to, and ordinary text generation asks it to do one thing at a time. Speculation is a way of manufacturing the parallel work the card wanted all along, by guessing what the work will be and checking the guess.

That also explains why the idea spread the way it did, from a Google translation paper to a weekend demo on a Mac to a dropdown in a desktop app in under two years. It needed no new models and no retraining, only a second file and a comparison rule, and it attacked the one bottleneck that every size of model and every size of GPU shares. If you have read the Inside the GPU series, you have met the memory wall over and over. Speculative sampling is the clearest example I know of what happens when someone stops trying to knock the wall down and finds a way to carry more through the door each time it opens.

Sources