Everybody Quotes the Speedup. Nobody Shows You the Knob
Speculative decoding sounds like a free lunch, which should always make you check your wallet. A tiny draft model guesses several tokens ahead, the big model checks them all in one pass, and every correct guess is basically free. Papers cite 2x to 3x and you nod along in the meeting, right up until you're the one implementing it and hit the question nobody put on the slide.
How long should the draft go before you're just burning cycles for style points? What acceptance rate do you actually need before this pays for itself? At what point does a longer draft start losing money because the big model keeps rejecting the tail end of it like a bouncer at a club it doesn't recognize you at?
speculabench answers those with no GPU and no weights, so you can find out you were wrong for free instead of on a production invoice. It models the accept and reject dynamics the way the actual papers do (Leviathan et al. and Chen et al., 2023) and reports the one number you care about: real speedup, measured in units of the expensive forward pass you're trying to avoid.
Turn the Knobs, Watch the Number Move
Three inputs, and they are the whole story.
• Agreement: how often the draft model matches the target. This is the single biggest lever on speedup, and it is the one number most teams never actually measure before shipping.
• Draft length: how many tokens to propose per round. Too short wastes the verification pass. Too long, and the tail gets rejected more than it gets accepted.
• Cost ratio: how expensive the draft pass is relative to the target. A slow draft model quietly eats the savings from every token it gets right.
The Sweet Spot Is a Real Curve, Not a Guess
Sweep draft length at a fixed 80% agreement and watch the tradeoff happen in real numbers: speedup climbs as the draft reaches further ahead, peaks, then falls as the rejected tail stops paying for itself. That turnover point is the entire engineering decision most teams make by vibes and a shrug in standup. Here it's a number you can print and staple to the PR.
The Number
At 90% draft agreement with a draft that costs a tenth of the target, speculative decoding hits 2.86x speedup. At 50% agreement, the same setup limps to 1.41x, and the honest lesson is that a bad draft model doesn't get a discount just because it's small. Small and wrong is still wrong.
"Not Hand-Wavy" Was Only True of Half the Model
I went back and checked what "faithful accounting, not a hand-wavy approximation" actually covered. The accept/reject bookkeeping, yes, that's the real Leviathan/Chen math. But the draft model decides match or mismatch with an independent coin flip at every single position. Nothing about how a real draft model fails looks like that. It nails long, predictable spans in unbroken streaks, then misses in clusters on the hard spans, because whatever makes one token hard to guess usually makes the next few hard too. That's autocorrelation, and independent coin flips have none of it by construction.
So I built a second draft model, a two-state Markov chain with a "burstiness" knob, calibrated so its average match rate tracks the original's within half a percentage point at 20,000 tokens no matter the burstiness setting. Same accuracy on paper. Then I swept draft length on both models at an identical 80% agreement rate and identical seed.
The independent-coin-flip model says stop at draft length 6, for 2.48x. The bursty model, same nominal accuracy, says stop at draft length 8, for 2.89x, a meaningfully better outcome the original model never surfaces because long agreement streaks pack more accepted tokens into a single verification pass than independent draws ever let them. I checked this held on a second seed I hadn't touched while building the fix, evaluated exactly once: the sweet spot shifted again from 6 to 8, 2.49x to 2.94x. Same story.
The original model and benchmark are untouched, so the published i.i.d. numbers above still reproduce exactly. The bursty model lives in a separate, opt-in module instead of silently becoming the new default.
16 tests pass in CI across Python 3.9, 3.11, and 3.13, including both benchmarks on every push. The whole simulation is deterministic, so the number you get is the number I got, no cherry-picked seed hiding in a drawer somewhere.