Everybody Quotes the Speedup. Nobody Shows You the Knob
Speculative decoding sounds like a free lunch. A tiny draft model guesses several tokens ahead, the big model checks them all in one pass, and every correct guess is basically free. Papers cite 2x to 3x and you nod along, right up until you try to implement it and hit the question nobody put on the slide.
How long should the draft go before you are just wasting cycles? What acceptance rate do you actually need before this pays for itself? At what point does a longer draft start losing money because the big model keeps rejecting the tail end of it?
speculabench answers those with no GPU and no weights. It models the accept and reject dynamics the way the actual papers do (Leviathan et al. and Chen et al., 2023) and reports the one number you care about: real speedup, measured in units of the expensive forward pass you are trying to avoid.
Turn the Knobs, Watch the Number Move
Three inputs, and they are the whole story.
• Agreement: how often the draft model matches the target. This is the single biggest lever on speedup, and it is the one number most teams never actually measure before shipping.
• Draft length: how many tokens to propose per round. Too short wastes the verification pass. Too long, and the tail gets rejected more than it gets accepted.
• Cost ratio: how expensive the draft pass is relative to the target. A slow draft model quietly eats the savings from every token it gets right.
The Sweet Spot Is a Real Curve, Not a Guess
Sweep draft length at a fixed 80% agreement and you can watch the tradeoff happen in real numbers: speedup climbs as the draft reaches further ahead, peaks, then falls as the rejected tail stops paying for itself. That turnover point is the entire engineering decision most teams make by feel. Here it is a number you can print.
The Number
At 90% draft agreement with a draft that costs a tenth of the target, speculative decoding hits 2.86x speedup. At 50% agreement, the same setup only manages 1.41x, and the honest lesson is that a bad draft model is not free just because it is small.
7 tests pass in CI across Python 3.9, 3.11, and 3.13. The whole simulation is deterministic, so the number you get is the number I got.