Speculative Decoding Tested: Does a Draft Model Help?
Speculative decoding on a 24GB unified memory machine actually slows down inference. Tested with qwen2.5 0.5B and 1.5B draft models. 15 runs, 5 trials each.
Short answer
On a 24 GB unified-memory machine, speculative decoding did not help. With Ollama 0.12.8, a Qwen3 14B target at Q4_K_M, and a Qwen 2.5 draft model in either 0.5 B or 1.5 B size, the baseline (no draft) was the fastest configuration. The 0.5 B draft was 5% slower. The 1.5 B draft was 10% slower. [Measured] 5 trials per config, deterministic settings, 200 generated tokens per trial. The published 1.5–2.0x speedup on discrete GPUs does not transfer to a single-user interactive workload on M2.
If you are serving a local LLM to one user on an M-series Mac, leave speculative decoding off. The 50 MB the draft model saves you in latency is the 50 MB you spent loading it into memory.
Why I tested this
I had read half a dozen blog posts and papers saying speculative decoding is the recommended pattern for fast local LLM serving. The setup is clean: a small model proposes the next few tokens, the big model verifies them in one pass, the accepted prefix becomes the new context. Published numbers said 1.5x to 2.0x speedup on a 4090. Apple Silicon was the gap. I wanted to know if the M2 in my laptop got the same win or whether unified memory changed the math.
I expected the answer to be yes. The M2 has fewer cores than a 4090 but more memory bandwidth per watt, and speculative decoding is bottlenecked by serial token generation, not parallel math. The math said it should work.
The data said no.
The setup
Target model: qwen3:14b at Q4_K_M (9.3 GB on disk, the same model from the Q4 vs MXFP4 benchmark). Draft models: qwen2.5:0.5b (397 MB) and qwen2.5:1.5b (about 1 GB). Both drafts use the Qwen tokenizer so the draft and target share a vocabulary, which is the requirement for speculative decoding to work without a translation step.
Same prompt for every run: a 263-character code-completion task asking for a Python function merge_dicts(*dicts) that takes any number of dicts, returns a single merged dict with later values winning, and includes a type hint. Deterministic settings: num_predict=200, temperature=0.0, seed=42. Five trials per configuration. 15 total runs.
The test machine was the M2 24 GB MacBook Pro I have been using for every other Hard Numbers benchmark. Ollama 0.12.8, Metal backend, no other significant load during the test. [Measured]
The headline numbers
| Config | Median wall (s) | Median eval (s) | Median tok/s | vs baseline |
|---|---|---|---|---|
| No draft (baseline) | 50.0 | 50.1 | 4.0 | 1.00x |
| Draft: qwen2.5:0.5b | 53.4 | 52.8 | 3.8 | 0.95x |
| Draft: qwen2.5:1.5b | 56.5 | 55.9 | 3.6 | 0.90x |
Speculative decoding slowed down inference by 5% with the smaller draft and 10% with the larger one. The expected 1.5–2.0x speedup did not materialize. The bigger draft was slower than the smaller one, which is the opposite of what the technique is supposed to do. [Measured]
Why it didn't help
Three things are likely going on, and I cannot fully separate them from these numbers alone.
Memory bandwidth is the bottleneck, not compute. Speculative decoding works by running the draft model first (cheap) and then the target model over a batch of draft proposals (one forward pass instead of N). The win comes from amortizing the cost of reading the target model's weights from memory across multiple tokens. On a 4090 with HBM, the weights for a 14B model at Q4 are about 8 GB; reading them once for a batch of 8 draft tokens is 8/8 = 1 GB per token of memory traffic. On M2 with unified memory, the same weights are 8 GB but the bandwidth is much lower (around 100 GB/s on M2 vs 1 TB/s on a 4090). The amortization works, but the base cost is higher, so the win is smaller.
The acceptance rate is low. A 0.5B draft and a 14B target will not agree on most token predictions. The published 1.5–2.0x speedups come from draft models chosen specifically for the target (often from the same family, sometimes distilled from the target itself). Qwen 2.5 is a different generation than Qwen 3 — they share a tokenizer and an architecture name, but the weights are not close. I would expect a Qwen 3 1.7B draft (if it existed as an Ollama model) to land closer to the published speedup, but I didn't have one installed and didn't want to pull another 2 GB to test.
Single-batch interactive inference is the worst case for the technique. Speculative decoding shines when the target model is being asked to generate a long sequence in a batched serving setup — the per-step overhead disappears into the batch. For one user generating 200 tokens, every step is its own forward pass, and the draft model's setup overhead is never amortized. The numbers here are the use case for which most people on a laptop actually run a local LLM.
Per-trial detail
| Trial | No draft | 0.5 B draft | 1.5 B draft |
|---|---|---|---|
| 1 | 45.8 s | 52.8 s | 55.9 s |
| 2 | 49.0 s | 50.1 s | 85.7 s |
| 3 | 50.5 s | 49.9 s | 109.1 s |
| 4 | 50.1 s | 62.2 s | 46.2 s |
| 5 | 51.2 s | 58.0 s | 52.7 s |
| Median | 50.1 s | 52.8 s | 55.9 s |
| Std dev | 2.0 s | 5.5 s | 25.0 s |
The 1.5 B draft has visibly more variance — trials 3 and 4 are 109 s and 46 s, a 2x spread. I would have discarded trial 3 as a warmup-cache miss, but it was trial 3, well after the warmup pass. The variance is real, and the median is still slower than baseline. [Measured]
What I'd try differently
If you want to push past the baseline on M2, the most promising thing is a closer-matched draft. Three things would change the result:
- A draft from the same model family. A Qwen 3 1.7B or 0.6B draft, if Ollama publishes one, would have a much higher acceptance rate against the Qwen 3 14B target. The published 1.5–2.0x speedups come from this kind of pairing.
- Newer Ollama with MTP. Ollama v0.32.6 (August 2026) added MTP-based automatic speculative decoding that uses the model's own multi-token-prediction head. No separate draft model needed. I am on 0.12.8, so this benchmark does not apply to that path.
- A quantized draft. The 0.5 B model at Q4_K_M is the smallest draft that still has usable English. Pulling the 0.5 B at Q2_K would halve its memory footprint and might reduce the draft overhead enough to flip the result.
I would bet that approach 1 is the only one that gets to 1.5x. The other two are smaller wins.
What we did not test
- Batched serving. This is the worst case for the technique. A high-throughput vLLM-style server handling 8 concurrent users would show different numbers. I do not run that workload.
- Long-context generation. Speculative decoding is supposed to help more as the prompt grows, because the draft model gets to look at the same context. I used a 263-character prompt. A 10K-token prompt might tell a different story.
- A close draft. I tested Qwen 2.5 0.5B and 1.5B against Qwen 3 14B. A Qwen 3 1.7B (if it existed) or a distilled 0.5B (one trained specifically to predict Qwen 3 14B) would almost certainly change the answer.
- Other target models. I used Qwen 3 14B because it is the model I have on disk and it is representative. Mistral-Small 24B and gpt-oss 20B would also be worth testing, especially because gpt-oss 20B has MTP heads built in.
- Other inference engines. llama.cpp directly, MLX, and vLLM all implement speculative decoding differently. Ollama's Metal path is one of the harder paths to make it work on, because of the unified-memory quirks. vLLM on an M-series Mac is not yet a thing I would run in production.
How to reproduce this
The full benchmark, the JSON results, and the warmup script are in the spec-decode-m2 folder of the experiments repo linked at the bottom.
# Pull the models
ollama pull qwen3:14b
ollama pull qwen2.5:0.5b
ollama pull qwen2.5:1.5b
# Run the benchmark
python3 src/benchmark.py
The script runs 5 trials per config, prints the per-trial table, and writes results/spec_decode.json with the full per-trial numbers.
Reproduction footer
This article, the benchmark code, the JSON results, and the warmup script are all in the experiments repo. If you re-run on different hardware, please open a PR with the results — the goal of Hard Numbers is to make every claim falsifiable.
- Code: github.com/Pitambarmahato/hardnumbers-experiments/tree/main/spec-decode-m2
- Results JSON: same repo,
spec-decode-m2/results/ - This article: hardnumbers.dev/articles/speculative-decoding-tested-does-a-draft-model-help
FAQ
Does speculative decoding speed up local LLMs?
It depends on the hardware and the workload. On discrete GPUs with high memory bandwidth, published numbers show 1.5–2.0x speedup. On Apple Silicon M2 24 GB with single-batch interactive inference, the draft model's overhead is not amortized, and the technique is 5–10% slower than the baseline. [Measured]
Which draft model size is best for speculative decoding?
For the Qwen 2.5 vs Qwen 3 14B pairing we tested, the smaller draft (0.5 B) was less harmful than the larger one (1.5 B), but neither was a speedup. In setups where the technique does help, the optimal draft is usually 0.5–2 B and from the same model family as the target. Larger drafts add more overhead per step and only help if the acceptance rate scales with draft size. [Inferred]
Is speculative decoding supported in Ollama?
Yes, since at least v0.12.8. Pass draft_model in the /api/generate or /api/chat request body, or set OLLAMA_SPECULATIVE_MODEL in the server's environment. Newer versions (v0.32.6+, August 2026) added MTP-based automatic speculative decoding that does not require a separate draft model.
Why does my published benchmark show 1.5x but the M2 shows the opposite?
Three reasons. First, the M2 has much lower memory bandwidth than a 4090, so the per-token cost of reading the target model's weights is higher, and the amortization that the technique relies on works less well. Second, the published benchmarks almost always use a draft model from the same family as the target, which gives a much higher acceptance rate than Qwen 2.5 drafting for Qwen 3. Third, the published benchmarks usually run in batched serving, not single-stream interactive generation. On an M2 with a cross-family draft and a single user, the technique does not pay off. [Measured, Inferred]
Should I leave speculative decoding on or off?
If you are running Ollama 0.12.8 with a draft model from a different family than the target and you are serving a single user on Apple Silicon, leave it off. The 5–10% slowdown is the most likely outcome. [Measured] If you upgrade to Ollama 0.32.6+ and use the MTP-based automatic speculative decoding, the answer is different and outside the scope of this benchmark.
Does speculative decoding change the model output?
No. The technique is lossless. The target model verifies every proposed token with an acceptance rule that preserves its original output distribution. With temperature=0.0, the output is byte-identical to the baseline run. [Documented]
FAQ
Does speculative decoding speed up local LLMs?
It depends on the hardware and the workload. On discrete GPUs with high memory bandwidth, published numbers show 1.5–2.0x speedup. On Apple Silicon M2 24 GB with single-batch interactive inference, the draft model's overhead is not amortized, and the technique is 5–10% slower than the baseline. [Measured]
Which draft model size is best for speculative decoding?
For the Qwen 2.5 vs Qwen 3 14B pairing we tested, the smaller draft (0.5 B) was less harmful than the larger one (1.5 B), but neither was a speedup. In setups where the technique does help, the optimal draft is usually 0.5–2 B and from the same model family as the target. Larger drafts add more overhead per step and only help if the acceptance rate scales with draft size. [Inferred]
Is speculative decoding supported in Ollama?
Yes, since at least v0.12.8. Pass draftmodel in the /api/generate or /api/chat request body, or set OLLAMASPECULATIVEMODEL in the server's environment. Newer versions (v0.32.6+, August 2026) added MTP-based automatic speculative decoding that does not require a separate draft model.
Why does my published benchmark show 1.5x but the M2 shows the opposite?
Three reasons. First, the M2 has much lower memory bandwidth than a 4090, so the per-token cost of reading the target model's weights is higher, and the amortization that the technique relies on works less well. Second, the published benchmarks almost always use a draft model from the same family as the target, which gives a much higher acceptance rate than Qwen 2.5 drafting for Qwen 3. Third, the published benchmarks usually run in batched serving, not single-stream interactive generation. On an M2 with a cross-family draft and a single user, the technique does not pay off. [Measured, Inferre
Should I leave speculative decoding on or off?
If you are running Ollama 0.12.8 with a draft model from a different family than the target and you are serving a single user on Apple Silicon, leave it off. The 5–10% slowdown is the most likely outcome. [Measured] If you upgrade to Ollama 0.32.6+ and use the MTP-based automatic speculative decoding, the answer is different and outside the scope of this benchmark.
Does speculative decoding change the model output?
No. The technique is lossless. The target model verifies every proposed token with an acceptance rule that preserves its original output distribution. With temperature=0.0, the output is byte-identical to the baseline run. [Documented]