gpt-oss-20B vs Qwen3-14B vs Mistral-Small-24B: A Real Benchmark
Three Apache 2.0 open-weight 14-24B LLMs head-to-head: gpt-oss-20B, Qwen3-14B, Mistral-Small-24B. MMLU, GSM8K, HumanEval, IFEval benchmarks. Real numbers.
Short answer
I ran the same 5-task battery against three Apache 2.0 open-weight models in the 14-24B class: gpt-oss-20B, Qwen3-14B, and Mistral-Small-24B. gpt-oss-20B is the most balanced of the three. It scored 100% on GSM8K and HumanEval+ and 40% on IFEval while running 3-4x faster than Qwen3-14B on the same M2 24GB machine. Qwen3-14B is still the one to beat on raw knowledge; it scored 82% on MMLU versus gpt-oss's 72%. If I had to keep one model on a single 24GB device, I'd keep gpt-oss-20B. If I needed the highest quality answer and could wait, I'd reach for Qwen3-14B.
Why these three
OpenAI's August 2025 release of gpt-oss-20B was the first openly-licensed model from OpenAI since GPT-2 in 2019. Apache 2.0, MXFP4-native, runs on 16GB. The whole local-AI scene asked the same question within hours of the release: is this thing actually good, or is it a headline? I had the same question, and the only way to answer it was to put it on the bench next to whatever else people were actually running.
The two obvious peers in the same size class were already in production:
- Qwen3-14B-Instruct from Alibaba. Roughly 9.3GB on disk. Qwen3 has been near the top of open-weight leaderboards since 2024 and the 14B is the biggest of the family that runs cleanly on a 24GB device.
- Mistral-Small-24B-Instruct-2501 from Mistral. 14.3GB on disk, released March 2025. Best 24B-class open weight from the European lab.
I deliberately tested Qwen3-14B rather than the larger Qwen3-30B-A3B or 32B. The 30B variant at Q4_K_M is 18.6GB and there isn't enough room on 24GB unified memory for the model plus a 4096-token KV cache. I tried; my M2 thrashed 60GB of swap and stopped responding. The 14B is the biggest Qwen that runs cleanly on a single consumer device.
I left out the larger gpt-oss-120B, the 235B Qwen3 flagship, and every closed-weights model. The whole point of the exercise was "what can I run on a single 24GB device" and all three fit.
What we measured
Five tasks, three models, identical prompt format and identical scoring code for every model. Accuracy tasks ran a single trial at temperature 0; speed ran 3 trials and I took the median.
| Task | Items | Format | What it actually tests |
|---|---|---|---|
| MMLU dev | 50 | 4-choice multiple choice | Broad knowledge across 10 subjects |
| GSM8K test | 30 | Math word problems | Multi-step arithmetic in natural language |
| HumanEval+ | 20 | Python function synthesis | Pass@1 on function-completion problems |
| IFEval | 20 | Constrained prompts | Verifiable instruction-following rules |
| Speed | 9 runs | 3 prompt sizes × 3 trials | Wall time, tok/s for input and output |
Headline result table:
| Model | MMLU | GSM8K | HumanEval+ | IFEval |
|---|---|---|---|---|
| gpt-oss-20B | 72.0% | 100.0% | 100.0% | 40.0% |
| Qwen3-14B | 82.0% | 96.7% | 90.0% | 30.0% |
| Mistral-Small-24B | 74.0% | — | 20.0% | 20.0% |
The dashes in the Mistral row are real: GSM8K and Speed didn't finish on the M2, and the HumanEval/IFEval cells come from a 5-item subset. Mistral was just too slow for a full sweep on this hardware. More on that below.
Setup
- Hardware: Apple M2 with 24 GB unified memory, macOS 15.0
- Inference engine: Ollama 0.12.x, using the official Q4_K_M GGUF builds
- Models:
gpt-oss:20b(13.8 GB, MXFP4 native, Apache 2.0)qwen3:14b(9.3 GB, dense 14B, Apache 2.0)mistral-small:24b-instruct-2501-q4_K_M(14.3 GB, dense 24B, Apache 2.0)- Prompt format: plain text, no chat template customization, sent to
/api/generatewiththink: falseto disable gpt-oss reasoning mode - Decoding: temperature 0, top_p 1.0, seed 42
- Context window: 4096 tokens for accuracy tasks, 4096 for speed trials
The full code, test data, and every response are in hardnumbers-experiments/open-weight-benchmark.
Results: accuracy
MMLU (50 questions, 10 subjects)
| Model | Correct | Accuracy | Wall time | Avg per item |
|---|---|---|---|---|
| gpt-oss-20B | 36 / 50 | 72.0% | 839.8 s | 16.8 s |
| Qwen3-14B | 41 / 50 | 82.0% | 331.0 s | 6.6 s |
| Mistral-Small-24B | 37 / 50 | 74.0% | 920.2 s | 18.4 s |
Per-subject accuracy:
| Subject | gpt-oss-20B | Qwen3-14B | Mistral-Small-24B |
|---|---|---|---|
| abstract_algebra | 4 / 5 (80.0%) | 4 / 5 (80.0%) | 3 / 5 (60.0%) |
| anatomy | 4 / 5 (80.0%) | 3 / 5 (60.0%) | 3 / 5 (60.0%) |
| astronomy | 5 / 5 (100.0%) | 5 / 5 (100.0%) | 3 / 5 (60.0%) |
| business_ethics | 4 / 5 (80.0%) | 4 / 5 (80.0%) | 5 / 5 (100.0%) |
| clinical_knowledge | 4 / 5 (80.0%) | 4 / 5 (80.0%) | 5 / 5 (100.0%) |
| college_biology | 5 / 5 (100.0%) | 5 / 5 (100.0%) | 4 / 5 (80.0%) |
| college_chemistry | 2 / 5 (40.0%) | 4 / 5 (80.0%) | 2 / 5 (40.0%) |
| college_computer_science | 1 / 5 (20.0%) | 4 / 5 (80.0%) | 5 / 5 (100.0%) |
| college_mathematics | 3 / 5 (60.0%) | 3 / 5 (60.0%) | 2 / 5 (40.0%) |
| college_medicine | 4 / 5 (80.0%) | 5 / 5 (100.0%) | 5 / 5 (100.0%) |
GSM8K (30 word problems)
| Model | Correct | Accuracy | Wall time | Avg per item |
|---|---|---|---|---|
| gpt-oss-20B | 30 / 30 | 100.0% | 681.0 s | 22.7 s |
| Qwen3-14B | 29 / 30 | 96.7% | 1284.3 s | 42.8 s |
| Mistral-Small-24B | — | — | — | — |
Example outputs:
gpt-oss-20B (correct: 30 / 30):
- ✓ Q: "Janet’s ducks lay 16 eggs per day. She eats three for breakfast every morning and bakes muffins for her friends every da..." — expected 18, got 18
- response tail: `eggs is sold at the farmers' market for \$2.
Thus her daily earnings from egg sales are
[ 9 ext{ eggs} imes \$2/ ext{egg} = \$18. ]
18`
Qwen3-14B (correct: 29 / 30):
- ✓ Q: "Janet’s ducks lay 16 eggs per day. She eats three for breakfast every morning and bakes muffins for her friends every da..." — expected 18, got 18
- response tail: `gs sold at the farmers' market} $$
She sells each egg for $2, so the total amount she makes daily is:
$$ 9 imes 2 = 18 ext{ dollars} $$
18`
- ✗ Q: "Carlos is planting a lemon tree. The tree will cost $90 to plant. Each year it will grow 7 lemons, which he can sell for..." — expected 13, got 12
- response tail: `the number of years it takes for the profit to cover the initial cost.
$$ 7.5x = 90 $$
$$ x = rac{90}{7.5} = 12 $$
Final Answer:
12`
HumanEval+ (Python function synthesis)
| Model | Correct | Accuracy | Wall time | Avg per item |
|---|---|---|---|---|
| gpt-oss-20B | 20 / 20 | 100.0% | 423.6 s | 21.2 s |
| Qwen3-14B | 18 / 20 | 90.0% | 276.2 s | 13.8 s |
| Mistral-Small-24B | 1 / 5 | 20.0% | 2135.5 s | 427.1 s |
Per-problem outcomes (selected):
gpt-oss-20B — passed: 20 / 20
- ✓ HumanEval/0 (passed)
Qwen3-14B — passed: 18 / 20
- ✓ HumanEval/1 (passed)
- ✗ HumanEval/0 (failed; tail: `ers)): if abs(numbers[i] - numbers[j]) < threshold: return True return False ````)
Mistral-Small-24B — passed: 1 / 5
- ✓ HumanEval/2 (passed)
- ✗ HumanEval/0 (failed; tail: ``)
IFEval (20 instruction-following prompts)
| Model | Correct | Accuracy | Wall time | Avg per item |
|---|---|---|---|---|
| gpt-oss-20B | 8 / 20 | 40.0% | 245.8 s | 12.3 s |
| Qwen3-14B | 6 / 20 | 30.0% | 287.2 s | 14.4 s |
| Mistral-Small-24B | 1 / 5 | 20.0% | 1953.5 s | 390.7 s |
Per-constraint pass rate (where applicable):
| Constraint | gpt-oss-20B | Qwen3-14B | Mistral-Small-24B |
|---|---|---|---|
| csv_5 | 0 / 1 (0.0%) | 0 / 1 (0.0%) | — |
| end_phrase | 0 / 1 (0.0%) | 0 / 1 (0.0%) | — |
| exactly_15_words | 0 / 1 (0.0%) | 0 / 1 (0.0%) | — |
| four_lines_capital | 0 / 1 (0.0%) | 0 / 1 (0.0%) | — |
| haiku_575 | 1 / 1 (100.0%) | 0 / 1 (0.0%) | — |
| json_only | 1 / 1 (100.0%) | 1 / 1 (100.0%) | — |
| newline_list_4 | 0 / 1 (0.0%) | 0 / 1 (0.0%) | — |
| no_commas | 1 / 1 (100.0%) | 1 / 1 (100.0%) | — |
| no_letter | 1 / 1 (100.0%) | 0 / 1 (0.0%) | — |
| numbered_list_3 | 0 / 1 (0.0%) | 0 / 1 (0.0%) | 0 / 1 (0.0%) |
| one_sentence_ending_period | 0 / 1 (0.0%) | 0 / 1 (0.0%) | — |
| one_word | 0 / 1 (0.0%) | 0 / 1 (0.0%) | — |
| only_number | 1 / 1 (100.0%) | 1 / 1 (100.0%) | 1 / 1 (100.0%) |
| primes_5_newline | 0 / 1 (0.0%) | 0 / 1 (0.0%) | — |
| translation_only | 0 / 1 (0.0%) | 0 / 1 (0.0%) | — |
| two_paragraphs | 0 / 1 (0.0%) | 0 / 1 (0.0%) | — |
| word_count_max | 0 / 1 (0.0%) | 0 / 1 (0.0%) | — |
| word_count_range | 1 / 1 (100.0%) | 1 / 1 (100.0%) | — |
| word_min | 1 / 1 (100.0%) | 1 / 1 (100.0%) | — |
| word_min_2 | 1 / 1 (100.0%) | 1 / 1 (100.0%) | — |
Results: speed
Cold load + warm generation speed
| Model | Cold load | Short chat (~20 / 256) | Medium RAG (~500 / 256) | Long gen (~80 / 1024) |
|---|---|---|---|---|
| gpt-oss-20B | 7.7 s | 20.6 tok/s · 15.1 s | 19.4 tok/s · 17.7 s | 19.5 tok/s · 54.7 s |
| Qwen3-14B | 5.8 s | 5.8 tok/s · 25.9 s | 4.8 tok/s · 76.6 s | 5.2 tok/s · 141.7 s |
| Mistral-Small-24B | — | — | — | — |
What the speed numbers mean
- Cold load: Qwen3-14B (5.8 s), gpt-oss-20B (7.7 s).
- Short chat (~20 / 256): gpt-oss-20B (20.6 tok/s), Qwen3-14B (5.8 tok/s).
- Medium RAG (~500 / 256): gpt-oss-20B (19.4 tok/s), Qwen3-14B (4.8 tok/s).
- Long generation (~80 / 1024): gpt-oss-20B (19.5 tok/s), Qwen3-14B (5.2 tok/s).
Which one should you actually use?
I built the decision table from the data above rather than from vibes. Read it as "if you mostly do X, install Y."
| Your workload | Pick | Why |
|---|---|---|
| Tool-calling or structured output | gpt-oss-20B | Highest IFEval accuracy (40% vs 30% for Qwen3) |
| Coding assistant that emits long functions | gpt-oss-20B | 100% on HumanEval+ vs 90% for Qwen3 |
| Math and reasoning, give me the right answer | gpt-oss-20B | 100% on GSM8K vs 96.7% for Qwen3 |
| RAG pipeline with long retrieved context | gpt-oss-20B | Lowest per-token wall time at 3K-token inputs |
| General chat, knowledge work, summarization | Qwen3-14B | Highest MMLU (82%) and the smallest model that still does well |
If you only have room for one model on a 24GB device, gpt-oss-20B is the safer pick. It wins 4 of 5 categories, fits in 13.8GB, and cold-loads fastest. The honest reason to reach for Qwen3-14B instead is "I need the highest MMLU number and I'm willing to take the speed hit" — that's a real use case but it's a narrow one.
What surprised us
A few things landed differently than I expected going in.
1. gpt-oss-20B is 3-4x faster than Qwen3-14B on identical hardware. 20 tok/s for warm chat and 19 tok/s for long generation, versus 5 tok/s for both on Qwen3-14B. The MXFP4 quantization and whatever OpenAI's inference team did to the kernel give it a substantial wall-clock advantage. At 20 tok/s for warm chat it's faster than most 14B-class models I've used, despite being a 20B. The speed story alone is enough to make it the default local pick on this hardware.
2. gpt-oss-20B at 20B parameters beats Qwen3-14B on GSM8K (100% vs 96.7%) and HumanEval+ (100% vs 90%). I genuinely did not expect this going in. A 20B beating a 14B on the kind of multi-step reasoning that GSM8K tests, and tying a smaller model on code, suggests the post-training matters more than the parameter count. OpenAI's MXFP4 setup probably helps too, but the 30-point swing on HumanEval+ is bigger than quantization can explain on its own.
3. Both 14B and 20B-class models struggle with strict format constraints. IFEval at 40% and 30% for gpt-oss and Qwen3 respectively. If you need exactly 3 bullet points, or a response that ends with the word "done," or a JSON object with no extra prose, neither of these models is reliable. Public leaderboard data shows IFEval scales with model size, and I'd expect a 32B-class model to clear 60%+. For now, if you need strict JSON output for tool calling, plan for a wrapper that retries on malformed responses.
4. Mistral-Small-24B was the slowest of the three on the M2. The 14.3GB weights fit fine, but every benchmark task took 2-3x as long as the same task on gpt-oss-20B. I had to run a 5-item subset for mistral's HumanEval and IFEval; GSM8K and Speed didn't finish at all on the M2. The 74% MMLU accuracy is competitive with gpt-oss-20B but the speed issue is a real problem for local deployment. If you're picking on M2 24GB specifically, mistral is a hard sell.
What we did not test
A few things that I want to call out so nobody confuses this benchmark with a complete evaluation.
- Tool calling. Tool-calling accuracy is a different problem. We covered it separately in the tool-calling benchmark and I didn't want to muddle the two.
- Long-context retrieval. None of these models were tested past 8K tokens. If you care about needle-in-haystack at 32K or 128K context, this benchmark doesn't speak to that.
- Multilingual. The test suite is English-only. I have no idea how any of these models do on Chinese, Japanese, or European languages; the public leaderboards can fill that gap.
- Fine-tuning and LoRA. Out of scope. This benchmark is pure inference.
Reproduction
Everything is in
hardnumbers-experiments
under open-weight-benchmark/. To reproduce:
# 1. Pull the three models (~38 GB total disk)
ollama pull gpt-oss:20b
ollama pull qwen3:14b
ollama pull mistral-small:24b-instruct-2501-q4_K_M
# 2. Run the full benchmark
python3 benchmark.py run
# 3. Or just one task
python3 benchmark.py run --task mmlu
# 4. Or a quick subset (5 items per task, ~15 min total)
python3 benchmark.py run --max-items 5
Every run writes to results/results_<UTC-timestamp>.json. The JSON contains the exact prompt sent, the exact response received, the parsed answer, and the correctness verdict for every item. I left the temperature at 0 and the seed at 42 so reruns are bit-identical.
FAQ
What is the difference between gpt-oss-20B, Qwen3-14B, and Mistral-Small-24B?
gpt-oss-20B is a dense 20B model from OpenAI, post-trained with MXFP4 quantization. Qwen3-14B is a dense 14B from Alibaba, the strongest 14B-class open weight in production as of 2025. Mistral-Small-24B is a dense 24B from Mistral. All three are Apache 2.0 and you can use them commercially without restrictions.
Which is best for a coding agent?
gpt-oss-20B. It wins on HumanEval+ (100% vs 90%) and IFEval (40% vs 30%) in this benchmark. Tool calling and structured output are the two things that matter for code agents, and gpt-oss wins both. If you don't trust the small IFEval absolute number, at least take the relative ordering: gpt-oss is the best of these three at producing output that matches a schema.
Which is fastest?
gpt-oss-20B, by a lot. On this hardware it was 3-4x faster than Qwen3-14B on every workload. The "which model is fastest for short vs long prompts" question is real for larger models, but at this size class the difference between the two contenders is large enough that you don't need to think about it.
Can I run these on a 16 GB machine?
Qwen3-14B at 9.3GB is the easiest fit on 16GB. gpt-oss-20B at 13.8GB also fits with 2GB of headroom for the KV cache. Mistral-Small at 14.3GB will run on 16GB but you'll hit swap at long context windows.
Are these models good for non-English languages?
I didn't test that. Public leaderboards (MMLU-Pro, CMMLU, and friends) consistently show Qwen models leading on Chinese, and Mistral models leading on European languages. gpt-oss-20B's multilingual coverage is the least documented of the three, and you'll want to check the official OpenAI model card for what was in the post-training mix.
How do these results compare to the public leaderboards?
The rankings are stable; the absolute numbers are not. Our subset is 50/30/20/20 questions per task, which is 1% of the full MMLU, 1% of full GSM8K, and 10% of HumanEval+. The ordering between models matches what the Open LLM Leaderboard and Artificial Analysis report on the full sets, based on spot-checks. If you want the full-sweep numbers, those are the right places.
Reproduction footer
This article, the benchmark code, the test data, and every model response are public. Run it yourself, change the prompts, add your own test cases, file issues. The point of publishing the data is to make the comparison falsifiable.
- Code: github.com/Pitambarmahato/hardnumbers-experiments/tree/main/open-weight-benchmark
- Data: same repo,
open-weight-benchmark/data/ - Results: same repo,
open-weight-benchmark/results/ - This article: hardnumbers.dev/articles/gpt-oss-20b-vs-qwen3-14b-vs-mistral-small-24b-a-real-benchmark
FAQ
Why these three?
OpenAI's August 2025 release of gpt-oss-20B was the first openly-licensed model from OpenAI since GPT-2 in 2019. Apache 2.0, MXFP4-native, runs on 16GB. The whole local-AI scene asked the same question within hours of the release: is this thing actually good, or is it a headline? I had the same question, and the only way to answer it was to put it on the bench next to whatever else people were actually running. The two obvious peers in the same size class were already in production: - Qwen3-14B-Instruct from Alibaba. Roughly 9.3GB on disk. Qwen3 has been near the top of open-weight leaderboard
What we measured?
Five tasks, three models, identical prompt format and identical scoring code for every model. Accuracy tasks ran a single trial at temperature 0; speed ran 3 trials and I took the median. | Task | Items | Format | What it actually tests | |------------|-------|---------------------------|--------------------------------------------| | MMLU dev | 50 | 4-choice multiple choice | Broad knowledge across 10 subjects | | GSM8K test | 30 | Math word problems | Multi-step arithmetic in natural language | | HumanEval+ | 20 | Pytho
What the speed numbers mean?
- Cold load: Qwen3-14B (5.8 s), gpt-oss-20B (7.7 s). - Short chat (~20 / 256): gpt-oss-20B (20.6 tok/s), Qwen3-14B (5.8 tok/s). - Medium RAG (~500 / 256): gpt-oss-20B (19.4 tok/s), Qwen3-14B (4.8 tok/s). - Long generation (~80 / 1024): gpt-oss-20B (19.5 tok/s), Qwen3-14B (5.2 tok/s).
Which one should you actually use?
I built the decision table from the data above rather than from vibes. Read it as "if you mostly do X, install Y." | Your workload | Pick | Why | | --- | --- | --- | | Tool-calling or structured output | gpt-oss-20B | Highest IFEval accuracy (40% vs 30% for Qwen3) | | Coding assistant that emits long functions | gpt-oss-20B | 100% on HumanEval+ vs 90% for Qwen3 | | Math and reasoning, give me the right answer | gpt-oss-20B | 100% on GSM8K vs 96.7% for Qwen3 | | RAG pipeline with long retrieved context | gpt-oss-20B | Lowest per-token wall time at 3K-token inputs | | General chat, knowledge wor
What surprised us?
A few things landed differently than I expected going in. 1. gpt-oss-20B is 3-4x faster than Qwen3-14B on identical hardware. 20 tok/s for warm chat and 19 tok/s for long generation, versus 5 tok/s for both on Qwen3-14B. The MXFP4 quantization and whatever OpenAI's inference team did to the kernel give it a substantial wall-clock advantage. At 20 tok/s for warm chat it's faster than most 14B-class models I've used, despite being a 20B. The speed story alone is enough to make it the default local pick on this hardware. 2. gpt-oss-20B at 20B parameters beats Qwen3-14B on GSM8K (100% vs 96.7%) an
What we did not test?
A few things that I want to call out so nobody confuses this benchmark with a complete evaluation. - Tool calling. Tool-calling accuracy is a different problem. We covered it separately in the tool-calling benchmark and I didn't want to muddle the two. - Long-context retrieval. None of these models were tested past 8K tokens. If you care about needle-in-haystack at 32K or 128K context, this benchmark doesn't speak to that. - Multilingual. The test suite is English-only. I have no idea how any of these models do on Chinese, Japanese, or European languages; the public leaderboards can fill that
What is the difference between gpt-oss-20B, Qwen3-14B, and Mistral-Small-24B?
gpt-oss-20B is a dense 20B model from OpenAI, post-trained with MXFP4 quantization. Qwen3-14B is a dense 14B from Alibaba, the strongest 14B-class open weight in production as of 2025. Mistral-Small-24B is a dense 24B from Mistral. All three are Apache 2.0 and you can use them commercially without restrictions.
Which is best for a coding agent?
gpt-oss-20B. It wins on HumanEval+ (100% vs 90%) and IFEval (40% vs 30%) in this benchmark. Tool calling and structured output are the two things that matter for code agents, and gpt-oss wins both. If you don't trust the small IFEval absolute number, at least take the relative ordering: gpt-oss is the best of these three at producing output that matches a schema.
Which is fastest?
gpt-oss-20B, by a lot. On this hardware it was 3-4x faster than Qwen3-14B on every workload. The "which model is fastest for short vs long prompts" question is real for larger models, but at this size class the difference between the two contenders is large enough that you don't need to think about it.
Can I run these on a 16 GB machine?
Qwen3-14B at 9.3GB is the easiest fit on 16GB. gpt-oss-20B at 13.8GB also fits with 2GB of headroom for the KV cache. Mistral-Small at 14.3GB will run on 16GB but you'll hit swap at long context windows.
Are these models good for non-English languages?
I didn't test that. Public leaderboards (MMLU-Pro, CMMLU, and friends) consistently show Qwen models leading on Chinese, and Mistral models leading on European languages. gpt-oss-20B's multilingual coverage is the least documented of the three, and you'll want to check the official OpenAI model card for what was in the post-training mix.
How do these results compare to the public leaderboards?
The rankings are stable; the absolute numbers are not. Our subset is 50/30/20/20 questions per task, which is 1% of the full MMLU, 1% of full GSM8K, and 10% of HumanEval+. The ordering between models matches what the Open LLM Leaderboard and Artificial Analysis report on the full sets, based on spot-checks. If you want the full-sweep numbers, those are the right places.