gemma4-research/CORPUS_tool_calling_2026-05.md

Gemma 4 Tool-Calling: State of the Art, May 2026

Six-week update on top of the April 2026 corpus. Supersedes parts of CORPUS_tool_calling_format.md and GOTCHAS.md — read this first if you're reaching for those documents to debug a tool-calling problem.

Last updated: 2026-05-25. Based on Ollama + llama.cpp + vLLM issue tracker review, two academic papers from late 2025, and convergent empirical evidence from two independent production agents running gemma4:26b.

TL;DR

1. The April-2026 rule "think: false silently kills gemma4:26b in multi-turn tool-call loops" is wrong as written. The bug is parser-side in Ollama, not model-side. It triggers on the combination system prompt + think:false + tools — and affects gemma4:e4b too, not just 26B MoE. (ollama/ollama#15539)
2. Ollama has shipped material tool-calling fixes since 0.20.4. Current stable is v0.24.0 (2026-05-14). v0.20.6 explicitly improved Gemma 4 tool calling and parallel-streaming tool calls; v0.22.1 updated the Gemma 4 renderer for thinking and tool calling. If you're below 0.22.1 you're running on patched-but-incomplete state.
3. The dominant production failure mode is not "model can't call tools" — it's "model decides not to call a tool when it should." This is a research-characterized problem now, and the best fix in the literature is NOT prompt engineering — it's a lightweight probe on the model's own hidden states (see § Probe & Prefill).
4. think is not a single boolean rule. Setting it correctly requires reasoning about num_predict budget, tool-argument length, and prompt complexity together. The recipe is at the end.
5. No community fine-tune of Gemma 4 fixes tool-calling reliability as of late May 2026. If you need a robust local agentic small model today, the field still recommends Qwen3-Coder. Gemma 4 is workable but requires the harness work documented below.

---

1. Ollama: what's been fixed, what's still broken

Release cadence April–May 2026: 0.20.4 → 0.20.5 → 0.20.6 → 0.21.0 → 0.21.3 → 0.22.0 → 0.22.1 → 0.23.x → 0.24.0. (0.30.0 is a pre-release reorganizing llama.cpp consumption and adding MLX on Apple Silicon.)

Fixed

• v0.20.6: "Gemma 4 tool calling ability is improved and updated to use Google's latest post-launch fixes" + "improved parallel tool calling for streaming responses." First real fix for the streaming-drops-tool-calls bug noted in the April corpus. (release notes)
• v0.22.1: "Updated the Gemma 4 renderer for thinking and tool calling improvements." (release notes)
• PR #15467 (closed via merge): Added native tool-call parsers for gemma4, qwen3, qwen3-coder, cogito, deepseek3, functiongemma, lfm2, ministral, olmo3, qwen3vl. Gemma 4 is now in the "registered parser" list.

Still open (verify before assuming a bug is gone)

Issue	Status	What's broken	Workaround
#15539	open, assigned	system + think:false + tools produces raw JSON in content with trailing <channel|> token and empty tool_calls. Affects 26B AND e4b.	Set think: True, or remove the system prompt, or move to llama.cpp from source
#15719	open	gemma4:26b infinite tool-call loop on 0.20.6/0.20.7 only when called via LiteLLM proxy — direct Ollama works, 0.20.5 works	Skip the proxy, or pin Ollama version
#15497	open	OpenAI-compat streaming returns Function.Index: 0 for every tool call when model has no registered parser	n/a now for Gemma 4 (parser registered); affects custom models
#15315	reopened	Raw call:tool{...} text leaks out on e4b when arguments contain JS-style quotes/backticks	Strip / replace special chars in tool args before display

Re-reading the April baseline against this

The April corpus said the think: false failure was a 26B-MoE-specific defect — the model emits near-immediate EOS at decision turns. Issue #15539's diagnosis suggests this was actually the Ollama parser failing to extract a tool call from output that, on the wire, contained one. The model emitted the call; the framework returned tool_calls=[] and an empty content with a stray <channel|> token.

If you have a think: false 26B harness that's failing, the right next diagnostic is to remove the system prompt and re-test — if it fires reliably, you've reproduced #15539. The fix is either upgrade Ollama past 0.22.1 and re-verify, or move to llama.cpp built from source with the PRs in § 2 applied.

---

2. llama.cpp: what's been fixed

Resolved

• The <unused49> / channel-token loop across CUDA, ROCm, Vulkan, SYCL was a llama.cpp eval bug, not quantization. Fixed in b8691+. Old quants work fine on patched builds. (unsloth/gemma-4-26B-A4B-it-GGUF#2)
• Special tokens leaking into tool-call argument values (e.g., [<|"|>light<|"|>]) — closed via fix in early April. (ggml-org/llama.cpp#21316)
• Canonical "build from source for Gemma 4 tool calling" PR pair: #21326 (template)
  • #21343 (tokenizer). This combination is cited by multiple writeups as the working baseline.

Still open or unclear

• #22080: gemma-4-26B-A4B "generation stops abruptly mid-output" on multi-GPU llama-server with --reasoning-budget set. No root cause. May be the same family as Ollama #15539 surfacing in a different runtime.
• 31B + f16/f32 vision projector still produces <unused49> floods on current builds per one report — the eval-bug fix didn't cover the multimodal projector path.
• Inverted thinking-mode guard in Google's released chat template (models/templates/google-gemma-4-31B-it.jinja): when enable_thinking=false, the upstream template emits a closed empty thought block; when true, it fails to open <|channel>thought\n. Local-fork fix only as of writing. (CompleteTech writeup)

--jinja is still required. The chat template is the contract.

---

3. vLLM: usable, with sharp edges

vLLM ships an official Gemma 4 recipe and a --tool-call-parser gemma4 flag. One open caveat:

• #39392: <pad> token floods under concurrent requests (2/5 fail at concurrency 5; 0/5 sequential). Reporter suspects shared mutable parser state — not thread-safe. Workaround is a global serialization lock around the tool-call path.

vLLM also still suffers the heterogeneous-attention-head Triton fallback (LOW-severity entry in GOTCHAS.md) — throughput is well below what similarly-sized models get on the same hardware.

---

4. The think flag, properly characterized

The April corpus had two contradictory entries: "always think: false for single-turn JSON" and "never think: false for multi-turn tool loops because it kills 26B." Both pointed at the same symptom (empty content, no usable response) with different prescriptions. Field evidence from two independent agents running gemma4:26b makes the actual interaction clearer.

The two failure paths

Path A — think: True + num_predict too small. The hidden reasoning trace consumes the entire generation budget, leaving message.content="" and done_reason="length". Observed against num_predict=1024 with a complex system prompt + tools schema; thinking trace alone was reaching the cap.

Path B — think: False + (long tool args OR multi-turn agentic loop OR system prompt present). Model emits a tool call on the wire that the parser fails to extract, or emits near-immediate EOS at a decision turn. Surface symptom: tool_calls=[], empty content, sometimes a stray <channel|> token. Per Ollama #15539 this is parser-side; per the April bakeoff it also shows up in deep multi-turn loops where each decision turn is short.

The recipe

Loop shape	think	num_predict	Why
Single-turn structured JSON	False	≥ 2048	No tool-call parsing involved; thinking adds latency without benefit; small budgets are safe
Short tool args, ≤ 5-step loop	False	≥ 1024	Field-verified to fire reliably at temp ≤ 0.5 with explicit Trigger→Action prompt structure
Long tool args (image-gen prompts, file payloads, code blocks)	True	≥ 2048	Field-verified: think: False produces empty tool_calls on the same context where think: True produces a proper call
Deep multi-turn agentic (≥ 10 steps)	True	≥ 2048	Per April bakeoff: think: False produces silent EOS at decision turns

Safety floor: if you can afford the latency, think: True + num_predict: 2048 works for all shapes above. The reason to deviate is one of:

• Latency budget < ~3s per turn (thinking adds 1–2s)
• Predictability is critical (no hidden token generation)
• Schema-constrained output that excludes thinking tokens

Build a probe. Both agents that diagnosed this in production wrote a probe script that called the same Ollama endpoint with each flag setting against a known-good message state and asserted tool_calls != []. If you're flipping think, write the probe first — same-context comparisons are how the parser-side failure surfaces.

Reproducing the parser-side diagnosis

To verify whether your harness is hitting Ollama #15539 vs a different failure path:

# Same model state, four conditions
combos = [
    {"system": None,  "think": True},
    {"system": None,  "think": False},
    {"system": "...", "think": True},
    {"system": "...", "think": False},
]
for c in combos:
    response = ollama.chat(model="gemma4:26b", messages=msgs(c), tools=TOOLS,
                           think=c["think"])
    print(c, "→ tool_calls:", len(response["message"]["tool_calls"]))

If only system + think:False produces zero tool calls, you've reproduced the parser bug, not the model defect.

---

5. The "doesn't fire tools without explicit ask" pattern

This is the dominant production failure mode for small/mid agentic models as of May 2026 — the model is capable of using tools, calls them when explicitly told ("look this up", "search for"), and defaults to answering-from-training otherwise. Now characterized in the literature:

Quantified

• "To Call or Not to Call" measures under/over-calling across model families. Gemma3-27B over-calls (462 vs 297 optimal); Mistral3.1-24B severely under-calls. Gemma 4 not directly benched but the family signal is toward over-calling under reasoning-mode conditions. The "doesn't fire" observation in production likely reflects think: false runs where the model goes straight to answer without the reasoning step that would surface the tool-call decision.
• BiasBusters quantifies tool-selection bias driven by description metadata and position. Tools with edited descriptions get 10× more usage than identical tools with original descriptions (GPT-4.1, Qwen2.5-7B). The wording of your tool descriptions is itself tuning firing rate; this is not a marginal effect.

What actually moves the needle

Probe & Prefill (arxiv 2605.09252) — the most concrete fix in the recent literature, and the one most likely to apply to a Gemma-4-sized agent.

Core finding: tool-call decisions are linearly decodable from the model's pre-generation hidden states with AUROC 0.89–0.96 — substantially outperforming the model's own verbalized reasoning about whether to call a tool. The technique:

1. Collect ~900 examples labeled "should call tool" vs "should not."
2. Train a logistic regression on the final-layer hidden state at the pre-generation position. Trains in seconds on CPU. < 1 ms inference overhead.
3. At inference, if the probe says "should call," prefill a short steering sentence ("I'll use a tool for this. ") to bias decoding toward the tool path.

Demonstrated 48% reduction in unnecessary tool calls with 1.7% accuracy loss on Qwen3 (1.7/4/14/32B) and Llama (3.1-8B, 3.3-70B). Not tested on Gemma 4 — but the technique is model-agnostic and the hidden-state signal has held across every architecture they tried.

This is the only technique in the May 2026 literature that beats careful prompt engineering on this specific problem. Worth a serious experiment for any non-trivial Gemma 4 agent.

Prompt-engineering techniques that help (but won't fully solve)

• Imperative tool descriptions with activation triggers beat passive descriptions. "Call this when the user asks about current events" fires more reliably than "Search the web for current information." (BiasBusters' 10× finding is the upper bound; expect smaller effects in practice.)
• Trigger → Action contracts in the system prompt ("If the user attempts a translation → call score_vocab") outperform softer guidance. Field-verified to take a Gemma 4 26B agent from ~0% tool-firing at default temperature to ~80% at temp 0.4.
• Lower temperature (0.3–0.5) increases firing rate on agentic loops. Default Ollama temperature (~0.8) biases toward conversational mimicry of recent chat history.
• Reactive-mode nudge loop: if the model emits text without calling a tool on a turn where tools are available, append a system message ("Was a tool needed? If yes, call it.") and re-prompt. One agent in the field uses this in its autonomous-research loop but not its reactive chat loop; the reactive loop has the firing-rate problem the autonomous loop doesn't.

Benchmarking your fix

NVIDIA's When2Call (NAACL 2025) is the relevant eval: 1000 MCQs scoring "answer directly / call tool / ask clarifying / decline." Their finding: RPO consistently > SFT for training models to make this decision well. Useful as the harness even if you're not training, because it gives you a single number to track across prompt / parameter changes.

---

6. Two-stage tool routing — still alive and well

In-model selection has not caught up to two-stage routing for the "should-call" decision as of May 2026. The router pattern has shifted from "another LLM" to "small classifier":

• ToolCallVerifier: ModernBERT-based 0.1B token classifier. Recall 0.92, precision 0.95, F1 0.935. Designed for prompt-injection defense but the same pattern works as a "should-call" gate.
• vLLM Semantic Router (v0.1 Iris, Jan 2026): production framework for taxonomy-guided routing + confidence-cascade.
• Probe & Prefill (above) is the limit case: the "router" is a logistic regression on the main model's own hidden states. No separate model to host.

Practical recommendation: if you're under 5 tools and can't afford to train a probe, Trigger → Action prompting + temp 0.4 + reactive nudge loop will get you to ~70–80% firing rate. Beyond that, the probe technique is the lowest-cost concrete improvement.

---

7. Structured output: XGrammar-2 changes the math

XGrammar-2 (MLC blog 2026-05-04, arxiv 2601.04426) is the headline structured-output development of the year:

• New "Structural Tag" composable JSON protocol uniformly expressing OpenAI Harmony, tool calling, reasoning channels, and custom output structures
• Cross-grammar caching with ~50% structure reuse on 50-tool schemas
• Repetition state compression: 534ms → 5.37ms (100×)
• Batching + speculative-decoding support
• Default structured-generation backend for vLLM, SGLang, TensorRT-LLM, MLC-LLM as of March 2026

The per-token overhead is now low enough for production. Smaller models showed "substantial gains in output accuracy" on BFCL-V3.

Important distinction: structured output forces the format once you've decided to call a tool. It does not solve "decide whether to call a tool." For that, see § 5.

Ollama's native format schema field (not the broken format: "json") remains the right Ollama-native option; community reports it's more reliable than free-form JSON prompting but still leaks for some model families. No Gemma-4-specific reliability data yet.

---

8. New Gemma 4 variants — slim pickings

No community fine-tune released to date materially improves tool-calling reliability over base Gemma 4 26B-it. Notable releases since April:

Release	What	Tool-calling relevance
google/gemma-4-26B-A4B-it-assistant	0.4B drafter for speculative decoding	Generation speed only — not a tool-calling tune
nvidia/Gemma-4-26B-A4B-NVFP4	NVFP4 quant for Blackwell	Hardware target, not a tune
Jackrong/Gemopus-4-26B-A4B-it-GGUF	SFT for style consistency / Markdown structure	Explicitly disclaims tool calling: "known compatibility issues, not unique to this model"
HF blog / TRL fine-tuning recipe	Fine-tune gemma-4-E2B-it for tool calling on H100	Recipe, not a released model
Unsloth GGUFs	All sizes, plus fine-tuning guide	No agentic tune

If you need a robust local agentic small model today, the field still recommends Qwen3-Coder family. Gemma 4 is workable for agentic use with the harness work above, but it's not the path of least resistance for that workload.

---

9. What this means for an existing harness

A concrete checklist if you have a Gemma 4 26B agent in production and are debugging tool-call reliability:

1. Check your Ollama version. Below 0.22.1, upgrade. Below 0.20.6, you're missing the first round of Gemma 4 tool-call fixes entirely.
2. Write the four-condition probe from § 4 against a known-good message state. If only system + think:False zeros out tool_calls, you have #15539, not a model defect.
3. Set the recipe from § 4 by loop shape. The safe combo for almost all cases is think: True + num_predict: 2048.
4. Audit your tool descriptions for activation verbs. "Call this when…" beats "Search for…" by a non-marginal margin per BiasBusters.
5. Lower temperature to 0.4 if you're at default. This alone took one production agent from ~0% to ~80% firing on triggers.
6. Add a reactive-mode nudge loop: if tool_calls=[] on a turn where the user asked something factual, re-prompt with a system message asking whether a tool was needed.
7. If still under-firing, prototype Probe & Prefill on ~900 labeled turns from your own log. This is the lowest-effort high-impact technique in the current literature.
8. Consider Qwen3-Coder for the tool-heavy path if you're not committed to Gemma 4 for other reasons (vision, multilingual, licensing).

Contradictions with prior corpus, flagged

• GOTCHAS.md HIGH: "think: false Kills Gemma 4 26B in Multi-Turn Tool-Calling Loops" — partially superseded. The failure is real but is parser-side in Ollama (#15539), not model-side. Same fix (use think: True) but the root cause framing was wrong, and the failure also affects e4b, not just 26B.
• SYNTHESIS.md "Mandatory Ollama Settings": the single-turn "always think: false" rule is still correct for single-turn JSON pipelines. The multi-turn agent guidance needs the recipe from § 4 applied.
• GOTCHAS.md MEDIUM: "Tool Calling Broken in Ollama v0.20.0 Streaming" — partially fixed (v0.20.6 + v0.22.1). Re-test on current version before assuming streaming tool calls are still broken.
• GOTCHAS.md LOW: "<unused> Token Infinite Loop (Vulkan backends)" — fixed in llama.cpp b8691+.

Sources

Primary:

• ollama/ollama#15539
• ollama/ollama releases
• ggml-org/llama.cpp#21316, #21321, #22080
• vllm-project/vllm#39392

Research:

• "LLM Agents Already Know When to Call Tools" — Probe & Prefill
• "To Call or Not to Call"
• BiasBusters
• NVIDIA When2Call
• XGrammar-2 paper, MLC blog

Community:

• Tim Gregg: llama.cpp Gemma 4 thinking-prompt fix
• unsloth/gemma-4-26B-A4B-it-GGUF discussion #2 (unused49 root cause)
• google/gemma-4-26B-A4B-it discussion #10 (thinking-mode behavior)