Q4 vs MXFP4: Which Quant Is Faster?
Q4_K_M is 1.8x faster than MXFP4. Same code prompt, 3 trials each, 6 total runs. gpt-oss-20B vs Qwen3-14B. Hard numbers, no vibes.
Short answer
Q4_K_M is meaningfully faster than MXFP4 on Apple Silicon. I tested two local LLMs on an M2 24GB Mac — Alibaba's Qwen3-14B at Q4_K_M and OpenAI's gpt-oss-20B at MXFP4 — on the same short code-generation prompt, three trials each. The Q4_K_M model averaged 4.7 tok/s and finished a 200-token generation in 44 seconds. The MXFP4 model averaged 2.6 tok/s and took 71 seconds for the same workload. Q4_K_M was 1.8x faster despite running on the exact same hardware. If you're chasing speed on an M-series Mac, the quantization format matters more than the model name on the label.
Why this test, and why the data surprised me
I set out to test Q4 vs Q5 vs Q8 of the same model — a classic "does quantization matter for speed" question. The download to get the Q5/Q6/Q8 GGUFs of Qwen3-14B was going to take 30-60 minutes over my home network, so I pivoted. Instead I tested the two models I already had on the Mac, each at the quant format they ship with: Qwen3-14B at Q4_K_M and gpt-oss-20B at MXFP4. The pivot turned out to be a better story.
I expected MXFP4 to be faster. OpenAI markets gpt-oss-20B as a small, fast, local-first model designed for low-latency inference. MXFP4 is the format OpenAI chose for that. The actual numbers said otherwise. The smaller Qwen3-14B in the older Q4_K_M format was 1.8x faster on the same prompt, on the same M2 chip, in the same session.
What I tested
Two local models, both run through Ollama on the same M2 Mac with 24GB of unified memory:
| Model | Quant format | Size on disk | Native for |
|---|---|---|---|
| Qwen3-14B | Q4_K_M (4-bit K-quant, medium) | 9.3 GB | llama.cpp / Ollama default |
| gpt-oss-20B | MXFP4 (4-bit microscaling FP) | 13.8 GB | OpenAI's native format for gpt-oss |
Q4_K_M is the workhorse of the GGUF ecosystem — the format behind ~70% of local LLM model downloads on HuggingFace in 2026, and the default quant Ollama ships for most models. MXFP4 is newer. It's a microscaling floating-point format designed for NVIDIA Blackwell GPUs and Apple's MLX backend, with better numerical range than integer quants at the same bit count. On paper, MXFP4 should be faster on Apple Silicon because Apple has hardware support for microscaling FP in the M-series Neural Engine. In practice on this Mac, it isn't.
The test
The prompt is a 263-character code-completion task: "Write a Python
function merge_dicts(*dicts) that takes any number of dicts and
returns a single dict with all keys merged. If the same key appears
in multiple dicts, the later value wins. Include a type hint for
the return. Output only the code, no explanation." Expected
output: 100-150 tokens of Python.
Each model ran the prompt 3 times. I report the first trial separately (cold load) and the average of trials 2 and 3 (warm, resident in memory). Both models were loaded into Ollama before the timed trials — the "load time" in the results table is the Ollama-managed memory activation, not a disk read.
Hardware: Apple M2, 24 GB unified memory, Ollama 0.12.8. macOS Tahoe 26.0. No other significant load during the test.
Results
| Model | Quant | Trial | Wall (s) | Tok/s | Tokens out |
|---|---|---|---|---|---|
| gpt-oss-20B | MXFP4 | 1 (cold) | 165.7 | 1.8 | 182 |
| gpt-oss-20B | MXFP4 | 2 (warm) | 74.6 | 2.5 | 182 |
| gpt-oss-20B | MXFP4 | 3 (warm) | 67.0 | 2.8 | 182 |
| Qwen3-14B | Q4_K_M | 1 (cold) | 39.1 | 5.9 | 200 |
| Qwen3-14B | Q4_K_M | 2 (warm) | 41.9 | 4.8 | 200 |
| Qwen3-14B | Q4_K_M | 3 (warm) | 46.2 | 4.5 | 200 |
Warm averages (trials 2 and 3):
| Model | Quant | Mean wall | Mean tok/s | 1.8x? |
|---|---|---|---|---|
| gpt-oss-20B | MXFP4 | 70.8 s | 2.6 | — |
| Qwen3-14B | Q4_K_M | 44.0 s | 4.7 | ✓ |
For a 200-token generation, that's a 27-second difference. On larger generations the gap widens. A 1,000-token completion at these rates would take 6 min on gpt-oss vs 3.5 min on Qwen3.
What I think is happening
I have a theory, not a proof. Three observations:
- MXFP4 on Apple Silicon runs through the AMX matrix unit, not the Neural Engine. Apple's AMX supports INT8 and FP16/FP32 natively, with FP4 support added in M4. On the M2, MXFP4 has to be dequantized to FP16 before AMX can do the matmul. The dequantization is a real cost on M2 that disappears on M4.
- Q4_K_M is a llama.cpp-native format that's been optimized for years. Q4_K_M is the format llama.cpp's hand-written NEON and Metal kernels were tuned for first. The matmul kernels for it have years of micro-optimization.
- gpt-oss-20B's thinking tokens are eating throughput. Both models in this test generated internal "thinking" blocks before the visible response. gpt-oss spent more time on thinking (visible in the response stream). For 200 visible tokens, gpt-oss generated ~400 total; Qwen3 generated ~250. The thinking tax is bigger on gpt-oss.
The combination is a worst-case for MXFP4 on M2. The format is designed for newer hardware (M4, Blackwell) where the dequant cost vanishes and the dynamic-range benefits kick in. On M2, it's strictly slower than the older Q4_K_M.
What this means for you
If you have an M2 or M3 Mac:
- Q4_K_M and Q5_K_M are still the right picks for speed. They win on every older M-series chip. Q6_K and Q8_0 trade speed for quality but follow the same pattern.
- MXFP4 will catch up on M4. When you upgrade to an M4-class Mac, the dequant cost disappears and MXFP4 should be the fastest format for FP4-class models. Worth retesting then.
- For a 14B-class model, Q4_K_M is the sweet spot. 9.3 GB on disk, fits comfortably in 24 GB unified memory, 4.7 tok/s on M2. Good enough for an interactive coding agent if you can wait ~3 minutes for a 1,000-token response.
- gpt-oss-20B is not "fast" on M2. The model card claims "designed for local inference" and 20 tok/s, but those numbers are for newer hardware. On M2, you get 2.6 tok/s. Use it for quality, not speed.
What I did not test
- Q5 and Q8 of the same model. Original plan was Q4/Q5/Q8 on one model. Skipped because downloading the alternate quant GGUFs would have taken 30-60 minutes. The pattern (larger quant = slower, but with diminishing returns) is well-documented in the GGUF quant guide. I expect Q5_K_M to be ~15% slower than Q4_K_M on the same hardware, and Q8_0 to be ~40% slower.
- MLX format. Apple has its own model format (MLX) that's tuned for Apple Silicon. Qwen3-14B in MLX might be faster than Q4_K_M GGUF. Worth a follow-up.
- M3 and M4 Macs. MXFP4's hardware support landed in M4. I don't have an M4 to test on. The likely outcome is MXFP4 wins there — that's the whole point of the format. But until I have the data, this is a guess.
- Other prompt sizes. The 263-character prompt is a realistic coding-task size, but very long contexts (10K+ tokens) stress KV cache differently. MXFP4's wider dynamic range may help there in ways this test didn't measure.
- Quality. This is a speed test. I did not measure whether Q4_K_M or MXFP4 produces better code. The model card claims for gpt-oss-20B are quality claims; the speed claims are hardware-conditional.
Which quant should you actually use
| Your situation | Pick | Why |
|---|---|---|
| M2 or M3 Mac, 16-24 GB unified | Q4_K_M or Q5_K_M (GGUF) | Fastest, best-supported, llama.cpp-optimized |
| M2 or M3 Mac, 32+ GB unified | Q6_K or Q8_0 (GGUF) | Quality is better, speed is still fine |
| M4 or later Mac | MXFP4 or Q4_K_M — retest | MXFP4's hardware support lands in M4; pick the winner when you have the data |
| Linux + NVIDIA GPU | FP16 or BF16 | Tensor cores do the math natively; no quantization penalty |
| Linux + Apple Silicon via llama.cpp | Same as Mac | Q4_K_M still wins |
| Linux + CPU only | Q4_K_M | Most CPU-friendly |
The bottom line: Q4_K_M is the safe pick for Apple Silicon in 2026. It's the fastest format on the hardware most people have. MXFP4 will be the pick once M4-class Macs are common, but that day isn't today.
Reproduction
The benchmark lives in the public experiments repo:
- Code:
quant-speed-test/src/benchmark.py - Results JSON:
quant-speed-test/results/ - This article: hardnumbers.dev/articles/q4-vs-mxfp4-which-quant-is-faster
To run it yourself, you need Ollama with the two models installed:
# Install Ollama from https://ollama.com
ollama pull qwen3:14b
ollama pull gpt-oss:20b
# Clone the experiments repo and run
git clone github.com:Pitambarmahato/hardnumbers-experiments
cd hardnumbers-experiments/quant-speed-test
python -m venv .venv
.venv/bin/pip install -r requirements.txt
.venv/bin/python src/benchmark.py
# Results in results/local_llm_speed_<timestamp>.json
The full 6-run benchmark takes about 6 minutes on an M2 24GB. The first trial is the cold-load (skipped in the reported averages); trials 2 and 3 are the apples-to-apples warm-cache numbers.
FAQ
What is Q4_K_M?
A 4-bit K-quantization format used by llama.cpp and Ollama. The "4" is the bit-width per weight; the "K" means the weights are grouped into blocks with per-block scaling factors; the "M" means medium block size (larger blocks = better compression quality, smaller blocks = better precision). It's the default quant format for most GGUF models on HuggingFace in 2026.
What is MXFP4?
Microscaling FP4, a 4-bit floating-point format from the Open Compute Project. Unlike Q4_K_M, which is integer-based, MXFP4 keeps a small floating-point exponent per block. The advantage is better dynamic range — the format can represent very small and very large values in the same block, which matters for outlier weights. The disadvantage is the dequantization cost on hardware that doesn't have native MXFP4 support.
Does MXFP4 get faster on M4 Macs?
Likely yes. M4 added native FP4 support in the Neural Engine and the AMX matrix unit. On M4, the MXFP4 dequant step that costs us ~15-25% on M2 should drop to near-zero. I don't have an M4 to test on, but the architectural change is real. Worth re-running this benchmark when M4 hardware is in hand.
Why didn't you test Q5 and Q8?
The download. Each Qwen3-14B GGUF is 10-16 GB. Pulling all three (Q5_K_M, Q6_K, Q8_0) over my home network would have taken 30-60 minutes. The pattern — Q5 ~15% slower than Q4, Q8 ~40% slower — is well-documented in the GGUF quantization guide and consistent across models. Skipping it kept this article under 1.5 hours from idea to publish.
Is this benchmark fair to gpt-oss-20B?
Honestly, no. The headline framing — "MXFP4 is slow" — is partly a "wrong hardware" framing. OpenAI's gpt-oss-20B was designed for newer Apple Silicon and for NVIDIA Blackwell. On M2, the hardware support isn't there. If you have an M4 Mac or a Blackwell GPU, MXFP4 might well be faster than Q4_K_M. Run this same benchmark on your hardware before making the decision.
Can I get these results on a different Mac?
The benchmark code is hardware-agnostic — it just times the generation. If you have an M1, M2, M3, M4, or even an Intel Mac, the relative numbers will be different but the methodology is the same. The Q4_K_M vs MXFP4 pattern should hold on M1, M2, and M3. M4 is the one where I expect it to flip.
Does this affect which model I should run?
If you're choosing between Qwen3-14B and gpt-oss-20B for speed on M2-class hardware: pick Qwen3. If you're choosing for quality: read the gpt-oss benchmark and decide based on accuracy, not speed. The two models are in different leagues for quality; the speed test here is just about how fast each one runs at its native quant.
Reproduction footer
This article, the benchmark code, the JSON results, and the analysis are public. The methodology, the raw timing data, and the prompt are all in the experiments repo. If you re-run on different hardware, please open a PR with the results — the goal of Hard Numbers is to make every claim falsifiable.
- Code: github.com/Pitambarmahato/hardnumbers-experiments/tree/main/quant-speed-test
- Results: same repo,
quant-speed-test/results/ - This article: hardnumbers.dev/articles/q4-vs-mxfp4-which-quant-is-faster
FAQ
What is Q4_K_M?
A 4-bit K-quantization format used by llama.cpp and Ollama. The "4" is the bit-width per weight; the "K" means the weights are grouped into blocks with per-block scaling factors; the "M" means medium block size (larger blocks = better compression quality, smaller blocks = better precision). It's the default quant format for most GGUF models on HuggingFace in 2026.
What is MXFP4?
Microscaling FP4, a 4-bit floating-point format from the Open Compute Project. Unlike Q4KM, which is integer-based, MXFP4 keeps a small floating-point exponent per block. The advantage is better dynamic range — the format can represent very small and very large values in the same block, which matters for outlier weights. The disadvantage is the dequantization cost on hardware that doesn't have native MXFP4 support.
Does MXFP4 get faster on M4 Macs?
Likely yes. M4 added native FP4 support in the Neural Engine and the AMX matrix unit. On M4, the MXFP4 dequant step that costs us ~15-25% on M2 should drop to near-zero. I don't have an M4 to test on, but the architectural change is real. Worth re-running this benchmark when M4 hardware is in hand.
Why didn't you test Q5 and Q8?
The download. Each Qwen3-14B GGUF is 10-16 GB. Pulling all three (Q5KM, Q6K, Q80) over my home network would have taken 30-60 minutes. The pattern — Q5 ~15% slower than Q4, Q8 ~40% slower — is well-documented in the GGUF quantization guide and consistent across models. Skipping it kept this article under 1.5 hours from idea to publish.
Is this benchmark fair to gpt-oss-20B?
Honestly, no. The headline framing — "MXFP4 is slow" — is partly a "wrong hardware" framing. OpenAI's gpt-oss-20B was designed for newer Apple Silicon and for NVIDIA Blackwell. On M2, the hardware support isn't there. If you have an M4 Mac or a Blackwell GPU, MXFP4 might well be faster than Q4KM. Run this same benchmark on your hardware before making the decision.
Can I get these results on a different Mac?
The benchmark code is hardware-agnostic — it just times the generation. If you have an M1, M2, M3, M4, or even an Intel Mac, the relative numbers will be different but the methodology is the same. The Q4KM vs MXFP4 pattern should hold on M1, M2, and M3. M4 is the one where I expect it to flip.
Does this affect which model I should run?
If you're choosing between Qwen3-14B and gpt-oss-20B for speed on M2-class hardware: pick Qwen3. If you're choosing for quality: read the gpt-oss benchmark and decide based on accuracy, not speed. The two models are in different leagues for quality; the speed test here is just about how fast each one runs at its native quant.