Local LLM Tool Calling for AI Agents: Qwen3 14B vs Llama 3.2 3B on Apple Silicon

Short answer

On an Apple M2 with 24 GB unified memory, Qwen3 14B produces valid, schema-correct tool calls 90% of the time, and Llama 3.2 3B produces the right tool only 50% of the time — but the 3B is 4× faster end-to-end. If you want a local LLM to actually drive an agent, the 14B is the one that picks up the phone. The 3B is fast enough to be tempting but you will spend the saved time writing retry logic.

The question

AI agents are everywhere in 2026, and most of them need a tool-calling LLM at the core. The cloud APIs handle this well. The question is whether small local models — the kind that run on a laptop — can do the same job, with the same reliability.

The leaderboard tells you which model is "smarter" in some abstract sense. It does not tell you which one will actually pick the right tool out of six candidates and put the right types in the right slots. That is the only thing that matters for an agent.

Why this matters

If you are wiring an agent framework — LangChain, Pydantic AI, Smolagents, your own loop — the LLM is the one place that can break your agent without warning. A model that picks the wrong tool 50% of the time is not an "agent" — it is a flaky function call generator that someone has to wrap in retries, schema validators, and fallback logic until it becomes slower than just calling the API. [Editorial, based on observed results below.]

I ran Qwen3 14B and Llama 3.2 3B on a fixed set of ten agent tasks on an M2 laptop and counted what actually happened.

Methodology

Hardware

Apple M2, 8 cores (4 performance + 4 efficiency)
24 GB unified memory
macOS 15, Metal 3
No external GPU, no cloud API

Software

Ollama for local model serving (Metal backend)
Models served via Ollama's chat API
temperature: 0 (deterministic), num_predict: 512
HTTP transport via the local Ollama socket

Models

Qwen3 14B (Q4_K_M) — 9.3 GB on disk, 14.8B params
Llama 3.2 3B (Q4_0) — 2.0 GB on disk, 3.2B params

Both quantized to Q4. Same set of tools offered to each model. Same system prompt. Same temperature. The only thing that varied was the model.

Tools offered

Six plausible productivity-agent tools, defined in OpenAI function-calling JSON schema style (which Ollama also accepts):

Tool	What it does
`get_weather`	Current weather for a city
`search_articles`	Search the publication archive
`create_task`	Add a to-do item
`send_email`	Email a recipient
`lookup_user`	Find a user by email
`book_meeting`	Book a calendar event

Each tool had a real JSON schema with required fields, enums where appropriate (e.g. priority: ["low","medium","high"]), and type constraints (string / integer / array).

Workload

Ten prompts. Each prompt is one user request that should trigger exactly one tool call. Prompts vary in complexity — some are one-shot ("what is the weather in Tokyo"), some require multiple arguments ("book a 30-minute design review tomorrow at 2pm with sam@ and jordan@"), some include constraints ("limit to 5", "use Celsius").

Each prompt has an expected tool that I scored by hand. We then measure three things per call:

Parse OK — the model produced parseable JSON
Schema valid — the JSON has the right name and arguments conform to the tool's schema (correct types, required fields present, values in the enum if there is one)
Tool correct — the schema-valid call picked the expected tool (catches the case where the model picks a valid but wrong tool)

A "wrong tool but valid schema" is a real failure mode — the agent framework would call the wrong function, the user sees the wrong result, and you debug for an hour before realising the LLM just hallucinated a tool choice.

Metrics

Parse rate — % of calls where JSON could be extracted
Valid rate — % of calls that pass schema validation
Tool correct rate — % of calls that picked the expected tool
Wall time — full request-to-response time including model load amortization (measured per call, not per token)
Tokens/sec — generated tokens / generation time, reported by Ollama

Results

Per-task scorecard

#	Expected tool	Qwen3 14B	Llama 3.2 3B
1	get_weather	✓ get_weather	✓ get_weather
2	get_weather	✓ get_weather	✓ get_weather
3	search_articles	✓ search_articles	✗ schema (limit: "5" not 5)
4	create_task	✓ create_task	✓ create_task
5	create_task	✓ create_task	✗ returned "NONE"
6	send_email	✓ send_email	✓ send_email
7	lookup_user	✓ lookup_user	✓ lookup_user
8	book_meeting	✓ book_meeting	✗ schema (attendees as string, not array)
9	book_meeting	✗ no JSON	✓ wrong tool (chose create_task)
10	search_articles	✓ search_articles	✗ schema (limit: "5" not 5)

[Observed, single-run, M2 24GB, Ollama, August 2026]

Aggregate

Metric	Qwen3 14B	Llama 3.2 3B
Parse rate	90% (9/10)	90% (9/10)
Schema-valid rate	90% (9/10)	60% (6/10)
Tool-correct rate	90% (9/10)	50% (5/10)
Wall time, median	27.8 s	1.1 s
Wall time, p90	75.4 s	2.1 s
Tokens/sec, median	6.7 tok/s	26.6 tok/s
Total tokens out	2,332	306

[Observed]

The 3B is roughly 25× faster wall time on the median task, and generates ~4× more tokens per second. It is not slow — it is actually fast by absolute standards, and you can absolutely use it for a chat surface. But the tool correctness is the headline: the 3B picks the wrong tool half the time and violates the schema 40% of the time. The 14B picks the right tool 9/10. [Observed, editorial conclusion based on this sample.]

What I actually saw in the responses

The 3B's failure mode is consistent and predictable: it likes to render numbers as quoted strings. The schema says "limit": { "type": "integer" } and the 3B produced "limit": "5". This is not a one-off — task 3 and task 10 both hit it. It happens because the 3B is trained to produce natural language, and natural language is full of quoted numbers. The 14B does the right thing because it's a "thinking" model that goes through a planning step before emitting the call. [Observed]

The 3B also sometimes returns the word "NONE" when it can't decide. That is a perfectly reasonable behaviour for a chat model — it's saying "no tool applies" — but in an agent context it is a silent failure. You don't get an exception, you get a polite no-op.

The 14B's single failure (task 9: book_meeting) was different: it returned no parseable JSON at all. Looking at the raw response, it produced a long, chatty, un-prompted explanation of what it was about to do, and never got around to the JSON. This is the "thinking model burns its token budget" failure I covered in the M2 benchmark — it is the same pathology, in a different costume. [Observed]

Failure cases

3B "limit": "5" instead of 5 — this hit twice. The model treats the schema as a hint, not a contract. [Observed]
3B returned "NONE" on a "remind me to call the dentist" prompt. Arguably correct (creating a task from "remind me" is a stretch), but the model was offered a create_task tool — that's the right answer. [Observed, editorial: this is the right answer in agent context, and the 3B punted on it.]
3B picked create_task instead of book_meeting on a "book a meeting" prompt. The 3B understood the request was about scheduling, and it was — but it picked a less-specific tool. A schema-valid call to the wrong function is the worst kind of failure: you don't see it in tests because the JSON parses. [Observed]
14B returned a chatty preamble instead of JSON on the second book_meeting task. With num_predict: 512 the model was verbose before getting to the call, and the JSON didn't make it out. With a higher num_predict (1024+) this would have been fine. [Observed]

Production considerations

Reliability matters more than speed for the agent decision step. The 3B at 50% correctness means half your agent runs call the wrong function. Wrapping that in retries doubles your average latency and still leaves the 25% schema-fail rate. The 14B at 90% correctness is a thing you can ship. [Editorial, based on observed data.]
The 3B is a great drafting model. For "summarize this article into 3 bullet points" or "rewrite this in a friendlier tone", the 3B at 26 tok/s is a great experience. The 14B at 6.7 tok/s feels sluggish for casual chat. Match the model to the task. [Inferred]
Run the 14B for tool selection, the 3B for everything else. This is the multi-model agent pattern: one model classifies and picks the tool, another model handles the chat surface and the summarisation. You get the reliability where it matters and the speed where it doesn't. [Editorial]
Type strictness in your schema definitions matters more than you think. If you can express your tool arguments as enums or constrained values, the 3B will get it right more often. Free-form string fields are where it drifts. [Inferred, based on observed "limit": "5" failure pattern.]
Set num_predict higher for thinking models. The 14B's single failure was a budget issue, not a competence issue. [Observed]

Limitations

Sample size of 10. This is a directional study, not a significance-tested benchmark. A real eval suite would have 100+ tasks per category and 3-5 runs per task to average over thermal noise. [Inferred]
Single machine, single OS. M2, 24 GB, macOS 15. The 3B is particularly sensitive to memory pressure — on a machine with less RAM or a different Ollama version, the numbers would shift. The 14B is bottlenecked on the 14B's context window, not on the hardware. [Inferred]
temperature: 0. Tool calling at non-zero temperature would have more variance. Real agents often run at 0; some run at small positive values. The numbers here are the "best case" for each model. [Documented]
Schema was small. Six tools, simple types. Real agents have 20-50 tools with nested objects and arrays. The 3B's failure rate will likely scale worse with schema complexity. [Inferred]
No multi-turn. I tested single-shot tool selection. Real agents do follow-up calls based on tool results, plan, retry, etc. That's a different experiment. [Documented]

Conclusion

For tool-calling on a laptop, Qwen3 14B is the only one of these two I'd trust to run an agent unsupervised. Llama 3.2 3B is demonstrably faster, but its 50% tool-correctness and 60% schema-validity rates mean an agent built on it is going to misfire on roughly one in two real requests. That's not an agent, that's a coin flip.

What I'd ship, in order:

Production agent with a local LLM → Qwen3 14B class, accept the latency, write good schemas.
Casual chat, summarisation, drafting → Llama 3.2 3B class, the speed is a real feature.
Multi-model routing → 14B picks the tool, 3B does the rest, best of both.

The smaller model is the better choice for latency-sensitive local workloads on M2 — but "latency-sensitive local workload" is not "agent." Pick the right model for the job. [Editorial conclusion based on observed data.]

FAQ

Q: Why these two models specifically? A: They were already on the machine from the previous M2 inference benchmark, and they bracket the meaningful local-LLM range. A 70B class model is a different study. [Inferred]

Q: What about tool-use fine-tunes like Hermes or ToolBench? A: Those are interesting and likely better at this exact task. We didn't have one available locally. The two we tested are general chat models being asked to do tool calling, which is the typical "drop a model into Ollama and try to make an agent" experience. [Documented]

Q: Does the system prompt matter? A: Hugely. The prompt I used was a single sentence: "respond with a JSON object of the form {...} and nothing else. If no tool fits, respond with NONE." A more elaborate prompt with examples (few-shot) would likely push the 3B's numbers up significantly. We didn't test that here. [Inferred]

Q: Would vLLM or MLX change the latency numbers? A: MLX on Apple Silicon is sometimes 10-20% faster than Ollama (which uses llama.cpp under the hood). The wall times here include Ollama's per-request overhead, so a leaner runtime would help. The correctness numbers are model-bound, not runtime-bound. [Inferred]

Q: What if I have more than 24 GB of RAM? A: An M4 Max with 64 GB unified memory can run a 32B at Q8 (~32 GB) with no swapping. The 32B class will be both faster and more reliable than the 14B. A 70B at Q4 (~40 GB) fits but leaves no headroom. We didn't test those — different study. [Inferred]

Q: Could I just use the OpenAI API and skip this whole question? A: Yes, and for many use cases that's the right answer. The local play makes sense when (a) you have a privacy or compliance constraint, (b) you're shipping a feature that runs on customer hardware, (c) you want zero marginal cost at inference time, or (d) you want to learn how agents actually work end-to-end without depending on a vendor. [Documented, editorial]