Speculative Decoding: The LLM Speed Hack That’s Too Good to Be True (But Isn’t)

Imagine you could run your largest language model twice as fast—same accuracy, same output quality, zero extra hardware—by adding a tiny, dumb model as a sidekick. That’s not a thought experiment. That’s speculative decoding, and it’s one of the most quietly impactful tricks in modern LLM deployment.

The Bottleneck Nobody Fixes

Standard LLM inference is agonizingly serial. You generate one token at a time, and every single token requires a full forward pass through the model. For a 7B-parameter model on modest hardware, that’s roughly 30–50 milliseconds per token. A 500-token response takes 15+ seconds. The model can’t “think ahead” because the next token depends on the one you just generated.

This is an architectural limitation, not a hardware one. You can’t batch your way out of it because each token is causally dependent on the previous one. You can’t prune the model without degrading quality. The sequential nature is baked in.

Except it’s not—not really.

The Core Insight: Draft, Don’t Generate

Speculative decoding exploits a simple observation: most tokens in natural language are easy to predict. A small, cheap model can guess the next word with surprising accuracy a surprising amount of the time. The large model only needs to step in when the draft gets it wrong.

Here’s how it works under the hood:

1. The drafter (a fast, small model—maybe 100MB) proposes a sequence of K tokens in one shot. This takes very little time.
2. The verifier (your big model) checks the entire sequence in a single forward pass. It rejects any token that doesn’t match its own distribution.
3. Where the draft is correct, the big model effectively validates K tokens at once—a free speedup.
4. Where the draft is wrong, the big model corrects from that point onward, so you never output a bad token.

The magic? The verification pass costs about the same as generating one token normally. If the drafter gets K tokens right on average, you get a ~K× speedup on inference time, while producing exactly the same output distribution as the original model.

Why This Works So Well

It shouldn’t be as effective as it is. The draft model is orders of magnitude smaller. But language has structure. Common words, frequent patterns, and predictable syntax mean the drafter succeeds most of the time.

Typical results with a well-tuned small draft model:

• Acceptance rate: 60–85% of drafted tokens pass verification
• Average draft length: 3–5 tokens per verification round
• Real-world speedup: 1.8× to 2.5× with zero quality loss

The cost? Two models loaded in memory instead of one. For most production deployments, that’s negligible compared to the throughput gain.

A Concrete Example

Say you’re running a 13B parameter model. Without speculative decoding, you generate one token every 40ms. For a 300-token response: 12 seconds.

With a 100M-parameter drafter (runs in ~5ms per draft) and a typical acceptance rate of 70% with draft length 4:

• Each verification round costs 40ms + 5ms = 45ms
• Each round produces ~3.3 accepted tokens on average (4 × 0.7 + correction tokens)
• 300 tokens requires roughly 90 rounds
• Total time: 90 × 45ms = 4.05 seconds

That’s a 3× speedup—and the output is mathematically indistinguishable from the standard generation.

Where the Cracks Show

Speculative decoding isn’t free money in every scenario. The technique shines on:

• Latency-sensitive applications: chatbots, real-time assistants
• Long responses: more tokens = more drafts = more savings
• Deterministic sampling: greedy decoding or low-temperature generation where the draft model’s predictions align best

It struggles with:

• High-entropy generation: creative writing, code generation, or very diverse output domains where the small model guesses poorly
• Hardware-constrained deployments: if you barely fit one model in VRAM, adding a second is non-trivial
• Batch inference: when processing many queries simultaneously, the serial bottleneck is less painful because you can batch

Implementation in Practice

You don’t need to train anything. Hugging Face Transformers added speculative decoding support in 2024. You grab any small model that shares the same tokenizer as your big model, set a draft_model parameter, and go.

The workflow is dead simple:

from transformers import AutoModelForCausalLM, AutoTokenizer

big_model = AutoModelForCausalLM.from_pretrained("big-model")
draft_model = AutoModelForCausalLM.from_pretrained("tiny-draft-model")
tokenizer = AutoTokenizer.from_pretrained("big-model")

# That's it. Inference runs 2x faster with zero code changes.
outputs = big_model.generate(
    input_ids,
    draft_model=draft_model,
    speculative_decoding=True
)

The Bigger Picture

Speculative decoding is part of a quiet shift in LLM optimization thinking. The community spent 2023 chasing quantization, pruning, and distillation—all methods that trade quality for speed. Speculative decoding inverts the trade-off: you keep the quality and borrow a cheap model’s speed.

It’s not a silver bullet. For some workloads, the overhead of running two models cancels out the gain. But for the common case—single-query inference on conversational models—it’s the closest thing to a free lunch in deep learning today.

Your next interaction with a chatbot might be running speculative decoding right now, and you’d never notice. That’s the point.