Understanding Speculative decoding in Transformers
Generating one token at a time can be very time consuming. Speculative decoding allows generating multiple tokens in advance. How does it work though?
An LLM model does not need to run one forward pass per token. It can run a single forward pass over an entire sequence of guesses, and that pass needs exactly the same weights that a single token would have needed. It does more arithmetic, but arithmetic was never what limited the speed. The real limit was the speed of moving the data in memory, and the guesses add almost no data movement.
To understand this deeply, we’ll go step by step.
A quick catch-up
LLMs read tokens. An algorithm cuts text into chunks of roughly four characters, so “The quick brown fox” becomes four tokens: The, quick, brown, fox. Each token is stored as an integer ID from a fixed vocabulary. A typical model has 50,000 to 200,000 of those IDs.
The model itself is a large collection of numbers called weights. When someone says they are running a 7B model, they mean seven billion of those numbers. Each weight is stored as an fp16 number, a 16-bit format, so each weight takes two bytes. The whole set needs about 14 GB of memory.
For every token the input passes through a stack of transformer blocks, and each block is mostly doing matrix multiplication of input with these weights. At the end of the stack, the model produces one raw score for every entry in the token vocabulary. Those scores are called logits. One full run of the input through this stack is a forward pass.
Logits are not probabilities. Probabilities should add up to 1, but the model can return any arbitrary score. A SoftMax function converts them into a probability distribution, where every value is positive and all values add up to one. Then we pick a token from that list. You can pick the highest score (Greedy decoding) or you can choose a high scoring random token (you can tune the randomness using the temperature setting of a model).
The model reads “The quick brown”, scores every possible next token, turns the scores into probabilities, and samples “fox”.
Why generation must go one token at a time
Generation is slow because the model cannot start the next step until it knows the previous token. That token does not exist until the forward pass is done, so the forward passes become a chain of dependencies. This is what people mean by autoregressive generation. Each output becomes the input for the next step.
There is one important optimization we can do here, and it is called the KV cache. At every step we add a new token, and it needs to “attend” to all previous tokens. For “The quick brown”, we have already calculated self-attention for all words with all previous words.
The first token attends to nothing, the second attends to the first, and the third attends to the first two.
Now the next token, “fox”, has to attend to all previous tokens, and that is the only new thing. We can reuse all previous values. We only need to calculate the attention of “fox” to the previous tokens.
So “fox” attends to “the”, “quick” and “brown”. The cache makes each step cheaper, but it cannot start the next step sooner.
Each pass needs the token from the pass before it. The KV cache saves work on old tokens, but it cannot start the next pass sooner.
A forward pass has two separate costs
Typically, when people describe inference, they look at overall throughput. But in fact there are two different hardware limitations, and they are measured in different units.
The first metric is arithmetic throughput. This is measured in units of floating point operations per second, or FLOPs. One multiply operation and one add operation together count as two FLOPs. A datacenter GPU like the H100 can do about 1,000 trillion FLOPs per second at 16-bit precision, which is written as 1,000 TFLOP/s.
The second metric is memory bandwidth, which is measured in bytes per second. This is the rate at which the chip can read data from its main memory. The same H100 can achieve a transfer rate of around 3.35 trillion bytes per second, or 3.35 TB/s.
The weights live in main memory, and the chip can only hold a small piece of them at a time. So before the model can use a weight, that weight has to travel from memory to the GPU. Every new token requires the full set to travel again.
Let us compare the two demands that one token makes on the hardware. The token requires one use of each weight, so the chip moves two bytes and does two FLOPs, resulting in a ratio of one FLOP per byte moved.
But the hardware is designed to support something like 300 FLOPs per byte moved, which is peak throughput divided by bandwidth, so 1,000 trillion / 3.35 trillion = ~300.
In plain terms: the GPU calculates much faster than the memory can send data. So the GPU is almost always idle, and it waits for the next bytes. The speed of the data supply sets the limit, not the speed of the calculation. This is what memory bound means.
This ratio is called arithmetic intensity, and the crossover point is called the ridge point. Below the ridge point the GPU finishes its work and then waits for data. Above it, memory keeps up and arithmetic becomes the limit. The figure below shows the shape.
Performance grows linearly from one token to the ridge point at 300. Above the ridge it flattens, because memory keeps up and the arithmetic units are the limit.
Generating one token sits at an intensity of 1, against a ridge point of 300. It is three hundred times below the crossover, which means the GPU spends almost all of its time waiting.
This is the property that allows speculative decoding to work. If the chip is idling on data, then extra arithmetic on the incoming data does not cost much.
One forward pass has two costs: moving the weights and doing the arithmetic. The slower one sets the pace.
Doing the arithmetic for a 7B model
The ratio tells us the shape of the problem. The actual times tell us the size. The numbers make the argument easier to trust, so let us use them.
Cost of one token for a 7B model on an H100. Moving the weights takes about 300 times longer than the arithmetic.
The important consequence is that the chip has far more arithmetic capacity than a single token can use. The chip can do hundreds of times more arithmetic work without adding much time, provided the amount of data movement stays the same.
That is exactly what a batch of guessed tokens does.
Four tokens still read the weights once. The data time stays at 4.2 milliseconds, while the arithmetic grows from 0.014 milliseconds to 0.056 milliseconds.
The causal mask for parallel guess checking
The first piece was the memory cost. The second piece is the causal mask, and it arises out of the structure of attention.
A transformer applies a causal mask, so that position i is only allowed to look at positions < i. The model cannot see the future when it predicts, which keeps the prediction honest. The figure below uses P for the last token of the prefix and g1 to g4 for the four guesses.
The causal mask lets one pass score every guess. P predicts g1, g1 predicts g2, and g4 predicts one extra token.
Now do a single forward pass with P and the four guesses. Position P predicts what g1 should be. Position g1 predicts what g2 should be, and so on. Position g4 predicts one extra token.
Each position produces exactly the distribution it would have produced in a separate decoding step at that same point. The mask guarantees this, because it prevents the later tokens from influencing the earlier positions.
This is normally useless because the guesses do not exist yet at the time you are generating them. But suppose someone hands you the four guesses. Now the slots can all be filled, and all five predictions can be scored in one forward pass. The model has no way to tell that the tokens are guesses. From its point of view it is reading an ordinary sequence.
This is the core of the trick. The main model checks all four guesses in one forward pass, not four separate passes.
How speculative decoding works
And here’s the algorithm.
The first step belongs to the draft model. This is a much smaller model, usually ten to fifty times smaller than the main model, and its only job is to produce a guess. It generates K guesses for the same prefix in the normal slow way. If K is four, it might write “jumps over the sleepy”.
The second step uses the main model. It is given the prefix and the four guesses and makes a single forward pass. The earlier prefix tokens are already in the KV cache, so only the new positions need work. This pass gives five distributions: one for each guess, plus one for the token after the last guess.
The third step is where the comparison happens, and it must go from left to right. Take the first guess, “jumps”, and compare it with the distribution at P. If the main model agrees, keep the token and move to the next guess.
The fourth step is where the process stops. Suppose the first three guesses survive and “sleepy” is rejected. Everything from that point onward is thrown away, because those distributions were computed from a prefix that contained “sleepy”. Since that prefix turned out to be wrong, the model was answering a question that no longer applies.
The fifth step repairs the damage. Draw one token from the corrected distribution, the part where the main model gives a higher probability than the draft. That token is correct by construction, because it comes from the main model’s probabilities, not from the draft’s.
The sixth step is simply to continue. Add the accepted tokens to the prefix and begin again.
The figure shows the middle case: three guesses survive, and the fourth is replaced.
The draft writes four guesses. The main model keeps the first three, rejects the fourth, and adds one new token.
In the best case, all four guesses survive, and you get four tokens plus one bonus token from the main model. That is five tokens for roughly the latency of a single main model forward pass. In the worst case the first guess fails, you get one token, and the work of the draft was wasted. The draft work is not free, and we count it shortly.
Why the check is cheap and why the answer does not change
The verification step is often described as if it were expensive, but it is not. It operates on the five distributions the forward pass just produced, which is roughly 100,000 numbers per distribution, and those numbers are already in memory. It also needs a few random numbers.
The forward pass, by contrast, moves 14 GB of weights through memory. The check is a rounding error, and the forward pass happens only once per cycle, no matter how many guesses.
The check is also lossless, which is a stronger claim than it may sound. It means the output distribution is exactly the same as plain sampling from the main model, not merely close to it. Here p is the main model’s probability for a token, and q is the draft’s. The figure shows the rule that makes this true.
The check keeps the main model’s distribution. A guess that the draft overrates is sometimes rejected, and the gap is drawn again.
Where the draft proposes too little, the corrected distribution fills the gap. Where it proposes too much, the excess is refused. The two effects cancel exactly.
The draft overrates A. The check keeps the common part and redraws the excess, so the final distribution equals the main model’s distribution.
How much the method actually gains
The acceptance rate, written as α, is the probability that a single guess survives. The gain per step follows from α and K.
Each extra guess is less likely to survive, but the survivors still add up to more than one token per step.
In this example the speedup is about 2.4×. Published implementations report similar numbers, for example Leviathan et al. (2023) and Chen et al. (2023).
Why the guess count stays small
A natural question arises from the memory argument. If the chip can do 300 times more arithmetic for the same data, why not take 300 guesses?
The first reason is that the draft model is sequential too. To produce 300 guesses, it has to run 300 times, and each of those passes is memory bound as well. The draft cost grows in a straight line, and at some point it exceeds what the main model saves.
The second reason is that the probability of acceptance decreases as you go further. If you tried K = 12, the twelfth guess would survive with probability α¹² ≈ 0.07, so most of the work for the later guesses is wasted.
The third reason is attention. Each extra guess adds work in the attention step, and that work grows with the square of the block length. For four to eight guesses the cost is small, but very long blocks become expensive.
The fourth reason is batch size. A production server has many requests going on at the same time. They all use the same weights, so one weight read serves all of them. The arithmetic intensity is therefore roughly equal to the batch size. When the batch is 256, the intensity is about 256 FLOPs per byte, which is close to the ridge point. Extra FLOPs then cost real time. Speculative decoding is a latency tool for one user at a time, not a throughput tool for a packed server.
When it does not help
Two things can make the method slower: a poor draft, and too many guesses. A poor draft hurts the performance so if the draft does not match the main model, the acceptance rate falls and the method can end up slower than plain decoding. This is why drafts are usually distilled from the main model: the small model is trained to copy the main model’s outputs. Another option is a prediction head, a small extra output layer trained to guess several tokens ahead. Medusa and EAGLE are well-known examples for this.
The speedup rises with K, peaks near K = 6, and falls after that. Each extra guess adds less.
If you liked the article, please like, share, and subscribe for more articles like this. I also run 1:1 mentorship sessions on system design, backend engineering, and interview preparation. Book a session at mentoxis.com/mentors/parminder-poonian.
Ready to put this into practice?
Book a 1:1 mock interview with a FAANG engineer, or work through free interview questions with a live code editor.











