gemma4-research/CORPUS_tool_calling_2026-05.md

# Gemma 4 Tool-Calling: State of the Art, May 2026

> Six-week update on top of the April 2026 corpus. **Supersedes parts of
> `CORPUS_tool_calling_format.md` and `GOTCHAS.md`** — read this first if
> you're reaching for those documents to debug a tool-calling problem.
>
> Last updated: 2026-05-25. Based on Ollama + llama.cpp + vLLM issue tracker
> review, two academic papers from late 2025, and convergent empirical
> evidence from two independent production agents running `gemma4:26b`.

## TL;DR

1. **The April-2026 rule "`think: false` silently kills `gemma4:26b` in
   multi-turn tool-call loops" is wrong as written.** The bug is parser-side
   in Ollama, not model-side. It triggers on the **combination** `system
   prompt + think:false + tools` — and affects `gemma4:e4b` too, not just
   26B MoE. ([ollama/ollama#15539](https://github.com/ollama/ollama/issues/15539))
2. **Ollama has shipped material tool-calling fixes since 0.20.4.** Current
   stable is **v0.24.0** (2026-05-14). v0.20.6 explicitly improved Gemma 4
   tool calling and parallel-streaming tool calls; v0.22.1 updated the Gemma 4
   renderer for thinking and tool calling. If you're below 0.22.1 you're
   running on patched-but-incomplete state.
3. **The dominant production failure mode is not "model can't call tools" —
   it's "model decides not to call a tool when it should."** This is a
   research-characterized problem now, and the best fix in the literature
   is **NOT prompt engineering** — it's a lightweight probe on the model's
   own hidden states (see § Probe & Prefill).
4. **`think` is not a single boolean rule.** Setting it correctly requires
   reasoning about `num_predict` budget, tool-argument length, and prompt
   complexity together. The recipe is at the end.
5. **No community fine-tune of Gemma 4 fixes tool-calling reliability** as
   of late May 2026. If you need a robust local agentic small model today,
   the field still recommends Qwen3-Coder. Gemma 4 is workable but requires
   the harness work documented below.

---

## 1. Ollama: what's been fixed, what's still broken

Release cadence April–May 2026: 0.20.4 → 0.20.5 → 0.20.6 → 0.21.0 → 0.21.3 →
0.22.0 → 0.22.1 → 0.23.x → 0.24.0. (0.30.0 is a pre-release reorganizing
llama.cpp consumption and adding MLX on Apple Silicon.)

### Fixed

- **v0.20.6**: "Gemma 4 tool calling ability is improved and updated to use
  Google's latest post-launch fixes" + "improved parallel tool calling for
  streaming responses." First real fix for the streaming-drops-tool-calls
  bug noted in the April corpus.
  ([release notes](https://github.com/ollama/ollama/releases/tag/v0.20.6))
- **v0.22.1**: "Updated the Gemma 4 renderer for thinking and tool calling
  improvements."
  ([release notes](https://github.com/ollama/ollama/releases/tag/v0.22.1))
- **PR #15467 (closed via merge)**: Added native tool-call parsers for
  `gemma4`, `qwen3`, `qwen3-coder`, `cogito`, `deepseek3`, `functiongemma`,
  `lfm2`, `ministral`, `olmo3`, `qwen3vl`. Gemma 4 is now in the
  "registered parser" list.

### Still open (verify before assuming a bug is gone)

| Issue | Status | What's broken | Workaround |
|-------|--------|---------------|------------|
| [#15539](https://github.com/ollama/ollama/issues/15539) | open, assigned | `system + think:false + tools` produces raw JSON in `content` with trailing `<channel\|>` token and empty `tool_calls`. Affects 26B AND e4b. | Set `think: True`, or remove the system prompt, or move to llama.cpp from source |
| [#15719](https://github.com/ollama/ollama/issues/15719) | open | `gemma4:26b` infinite tool-call loop on 0.20.6/0.20.7 **only when called via LiteLLM proxy** — direct Ollama works, 0.20.5 works | Skip the proxy, or pin Ollama version |
| [#15497](https://github.com/ollama/ollama/issues/15497) | open | OpenAI-compat streaming returns `Function.Index: 0` for every tool call when model has no registered parser | n/a now for Gemma 4 (parser registered); affects custom models |
| [#15315](https://github.com/ollama/ollama/issues/15315) | reopened | Raw `call:tool{...}` text leaks out on e4b when arguments contain JS-style quotes/backticks | Strip / replace special chars in tool args before display |

### Re-reading the April baseline against this

The April corpus said the `think: false` failure was a 26B-MoE-specific
defect — the model emits near-immediate EOS at decision turns. **Issue
#15539's diagnosis suggests this was actually the Ollama parser failing
to extract a tool call from output that, on the wire, contained one.** The
model emitted the call; the framework returned `tool_calls=[]` and an
empty `content` with a stray `<channel|>` token.

If you have a `think: false` 26B harness that's failing, the right next
diagnostic is to **remove the system prompt and re-test** — if it fires
reliably, you've reproduced #15539. The fix is either upgrade Ollama past
0.22.1 and re-verify, or move to llama.cpp built from source with the PRs
in § 2 applied.

---

## 2. llama.cpp: what's been fixed

### Resolved

- **The `<unused49>` / channel-token loop across CUDA, ROCm, Vulkan, SYCL**
  was a llama.cpp eval bug, not quantization. Fixed in b8691+. Old quants
  work fine on patched builds.
  ([unsloth/gemma-4-26B-A4B-it-GGUF#2](https://huggingface.co/unsloth/gemma-4-26B-A4B-it-GGUF/discussions/2))
- **Special tokens leaking into tool-call argument values**
  (e.g., `[<|"|>light<|"|>]`) — closed via fix in early April.
  ([ggml-org/llama.cpp#21316](https://github.com/ggml-org/llama.cpp/issues/21316))
- **Canonical "build from source for Gemma 4 tool calling" PR pair**:
  [#21326](https://github.com/ggml-org/llama.cpp/pull/21326) (template)
  + [#21343](https://github.com/ggml-org/llama.cpp/pull/21343) (tokenizer).
  This combination is cited by multiple writeups as the working baseline.

### Still open or unclear

- **[#22080](https://github.com/ggml-org/llama.cpp/issues/22080)**:
  `gemma-4-26B-A4B` "generation stops abruptly mid-output" on multi-GPU
  llama-server with `--reasoning-budget` set. No root cause. May be the
  same family as Ollama #15539 surfacing in a different runtime.
- **31B + f16/f32 vision projector** still produces `<unused49>` floods on
  current builds per one report — the eval-bug fix didn't cover the
  multimodal projector path.
- **Inverted thinking-mode guard** in Google's released chat template
  (`models/templates/google-gemma-4-31B-it.jinja`): when
  `enable_thinking=false`, the upstream template emits a closed empty
  thought block; when true, it fails to open `<|channel>thought\n`.
  Local-fork fix only as of writing.
  ([CompleteTech writeup](https://complete.tech/blog/llamacpp-gemma4-thinking-prompt-local-fix/))

`--jinja` is still required. The chat template is the contract.

---

## 3. vLLM: usable, with sharp edges

vLLM ships an official Gemma 4 recipe and a `--tool-call-parser gemma4`
flag. One open caveat:

- **[#39392](https://github.com/vllm-project/vllm/issues/39392)**:
  `<pad>` token floods under concurrent requests (2/5 fail at concurrency
  5; 0/5 sequential). Reporter suspects shared mutable parser state —
  not thread-safe. Workaround is a global serialization lock around the
  tool-call path.

vLLM also still suffers the heterogeneous-attention-head Triton fallback
(LOW-severity entry in GOTCHAS.md) — throughput is well below what
similarly-sized models get on the same hardware.

---

## 4. The `think` flag, properly characterized

The April corpus had two contradictory entries: "always `think: false`
for single-turn JSON" and "never `think: false` for multi-turn tool loops
because it kills 26B." Both pointed at the same symptom (empty content,
no usable response) with different prescriptions. Field evidence from two
independent agents running `gemma4:26b` makes the actual interaction
clearer.

### The two failure paths

**Path A — `think: True` + `num_predict` too small.**
The hidden reasoning trace consumes the entire generation budget, leaving
`message.content=""` and `done_reason="length"`. Observed against
`num_predict=1024` with a complex system prompt + tools schema; thinking
trace alone was reaching the cap.

**Path B — `think: False` + (long tool args OR multi-turn agentic loop
OR system prompt present).** Model emits a tool call on the wire that the
parser fails to extract, or emits near-immediate EOS at a decision turn.
Surface symptom: `tool_calls=[]`, empty content, sometimes a stray
`<channel|>` token. Per Ollama #15539 this is parser-side; per the April
bakeoff it also shows up in deep multi-turn loops where each decision
turn is short.

### The recipe

| Loop shape | `think` | `num_predict` | Why |
|------------|---------|---------------|-----|
| Single-turn structured JSON | `False` | ≥ 2048 | No tool-call parsing involved; thinking adds latency without benefit; small budgets are safe |
| Short tool args, ≤ 5-step loop | `False` | ≥ 1024 | Field-verified to fire reliably at temp ≤ 0.5 with explicit Trigger→Action prompt structure |
| Long tool args (image-gen prompts, file payloads, code blocks) | `True` | ≥ 2048 | Field-verified: `think: False` produces empty `tool_calls` on the same context where `think: True` produces a proper call |
| Deep multi-turn agentic (≥ 10 steps) | `True` | ≥ 2048 | Per April bakeoff: `think: False` produces silent EOS at decision turns |

**Safety floor:** if you can afford the latency, `think: True` +
`num_predict: 2048` works for *all* shapes above. The reason to deviate
is one of:
- Latency budget < ~3s per turn (thinking adds 1–2s)
- Predictability is critical (no hidden token generation)
- Schema-constrained output that excludes thinking tokens

**Build a probe.** Both agents that diagnosed this in production wrote a
probe script that called the same Ollama endpoint with each flag setting
against a known-good message state and asserted `tool_calls != []`. If
you're flipping `think`, write the probe first — same-context comparisons
are how the parser-side failure surfaces.

### Reproducing the parser-side diagnosis

To verify whether your harness is hitting Ollama #15539 vs a different
failure path:

```python
# Same model state, four conditions
combos = [
    {"system": None,  "think": True},
    {"system": None,  "think": False},
    {"system": "...", "think": True},
    {"system": "...", "think": False},
]
for c in combos:
    response = ollama.chat(model="gemma4:26b", messages=msgs(c), tools=TOOLS,
                           think=c["think"])
    print(c, "→ tool_calls:", len(response["message"]["tool_calls"]))
```

If only `system + think:False` produces zero tool calls, you've reproduced
the parser bug, not the model defect.

---

## 5. The "doesn't fire tools without explicit ask" pattern

This is the dominant production failure mode for small/mid agentic models
as of May 2026 — the model is capable of using tools, calls them when
explicitly told ("look this up", "search for"), and defaults to
answering-from-training otherwise. Now characterized in the literature:

### Quantified

- **["To Call or Not to Call"](https://arxiv.org/html/2605.00737v1)**
  measures under/over-calling across model families. **Gemma3-27B over-calls
  (462 vs 297 optimal)**; Mistral3.1-24B severely under-calls. Gemma 4 not
  directly benched but the family signal is toward over-calling under
  reasoning-mode conditions. The "doesn't fire" observation in production
  likely reflects `think: false` runs where the model goes straight to
  answer without the reasoning step that would surface the tool-call
  decision.
- **[BiasBusters](https://arxiv.org/pdf/2510.00307)** quantifies
  tool-selection bias driven by description metadata and position. Tools
  with **edited** descriptions get **10× more usage** than identical tools
  with original descriptions (GPT-4.1, Qwen2.5-7B). The wording of your
  tool descriptions is itself tuning firing rate; this is not a marginal
  effect.

### What actually moves the needle

**Probe & Prefill** ([arxiv 2605.09252](https://arxiv.org/html/2605.09252v1))
— the most concrete fix in the recent literature, and the one most likely
to apply to a Gemma-4-sized agent.

Core finding: **tool-call decisions are linearly decodable from the
model's pre-generation hidden states with AUROC 0.89–0.96** —
substantially outperforming the model's own verbalized reasoning about
whether to call a tool. The technique:

1. Collect ~900 examples labeled "should call tool" vs "should not."
2. Train a logistic regression on the final-layer hidden state at the
   pre-generation position. Trains in seconds on CPU. < 1 ms inference
   overhead.
3. At inference, if the probe says "should call," prefill a short
   steering sentence ("I'll use a tool for this. ") to bias decoding
   toward the tool path.

Demonstrated **48% reduction in unnecessary tool calls with 1.7% accuracy
loss** on Qwen3 (1.7/4/14/32B) and Llama (3.1-8B, 3.3-70B). Not tested
on Gemma 4 — but the technique is model-agnostic and the hidden-state
signal has held across every architecture they tried.

**This is the only technique in the May 2026 literature that beats
careful prompt engineering on this specific problem.** Worth a serious
experiment for any non-trivial Gemma 4 agent.

### Prompt-engineering techniques that help (but won't fully solve)

- **Imperative tool descriptions with activation triggers** beat passive
  descriptions. "Call this when the user asks about current events"
  fires more reliably than "Search the web for current information."
  (BiasBusters' 10× finding is the upper bound; expect smaller effects
  in practice.)
- **Trigger → Action contracts in the system prompt** ("If the user
  attempts a translation → call `score_vocab`") outperform softer
  guidance. Field-verified to take a Gemma 4 26B agent from ~0%
  tool-firing at default temperature to ~80% at temp 0.4.
- **Lower temperature** (0.3–0.5) increases firing rate on agentic
  loops. Default Ollama temperature (~0.8) biases toward
  conversational mimicry of recent chat history.
- **Reactive-mode nudge loop**: if the model emits text without calling
  a tool on a turn where tools are available, append a system message
  ("Was a tool needed? If yes, call it.") and re-prompt. One agent in
  the field uses this in its autonomous-research loop but not its
  reactive chat loop; the reactive loop has the firing-rate problem
  the autonomous loop doesn't.

### Benchmarking your fix

[NVIDIA's When2Call](https://github.com/NVIDIA/When2Call) (NAACL 2025) is
the relevant eval: 1000 MCQs scoring "answer directly / call tool / ask
clarifying / decline." Their finding: **RPO consistently > SFT** for
training models to make this decision well. Useful as the harness even
if you're not training, because it gives you a single number to track
across prompt / parameter changes.

---

## 6. Two-stage tool routing — still alive and well

In-model selection has **not** caught up to two-stage routing for the
"should-call" decision as of May 2026. The router pattern has shifted
from "another LLM" to "small classifier":

- **[ToolCallVerifier](https://huggingface.co/llm-semantic-router/toolcall-verifier)**:
  ModernBERT-based 0.1B token classifier. Recall 0.92, precision 0.95,
  F1 0.935. Designed for prompt-injection defense but the same pattern
  works as a "should-call" gate.
- **vLLM Semantic Router (v0.1 Iris, Jan 2026)**: production framework
  for taxonomy-guided routing + confidence-cascade.
- **Probe & Prefill** (above) is the limit case: the "router" is a
  logistic regression on the *main model's own* hidden states. No
  separate model to host.

**Practical recommendation:** if you're under 5 tools and can't afford
to train a probe, Trigger → Action prompting + temp 0.4 + reactive nudge
loop will get you to ~70–80% firing rate. Beyond that, the probe technique
is the lowest-cost concrete improvement.

---

## 7. Structured output: XGrammar-2 changes the math

**[XGrammar-2](https://blog.mlc.ai/2026/05/04/xgrammar-2-fast-customizable-structured-generation)**
(MLC blog 2026-05-04, [arxiv 2601.04426](https://arxiv.org/abs/2601.04426))
is the headline structured-output development of the year:

- New "Structural Tag" composable JSON protocol uniformly expressing
  OpenAI Harmony, tool calling, reasoning channels, and custom output
  structures
- Cross-grammar caching with ~50% structure reuse on 50-tool schemas
- Repetition state compression: 534ms → 5.37ms (100×)
- Batching + speculative-decoding support
- **Default structured-generation backend for vLLM, SGLang, TensorRT-LLM,
  MLC-LLM** as of March 2026

The per-token overhead is now low enough for production. Smaller models
showed "substantial gains in output accuracy" on BFCL-V3.

**Important distinction:** structured output forces the **format** once
you've decided to call a tool. It does **not** solve "decide whether to
call a tool." For that, see § 5.

Ollama's native `format` schema field (not the broken `format: "json"`)
remains the right Ollama-native option; community reports it's more
reliable than free-form JSON prompting but still leaks for some model
families. No Gemma-4-specific reliability data yet.

---

## 8. New Gemma 4 variants — slim pickings

No community fine-tune released to date materially improves tool-calling
reliability over base Gemma 4 26B-it. Notable releases since April:

| Release | What | Tool-calling relevance |
|---------|------|------------------------|
| [`google/gemma-4-26B-A4B-it-assistant`](https://huggingface.co/google/gemma-4-26B-A4B-it-assistant) | 0.4B drafter for speculative decoding | Generation speed only — not a tool-calling tune |
| `nvidia/Gemma-4-26B-A4B-NVFP4` | NVFP4 quant for Blackwell | Hardware target, not a tune |
| [`Jackrong/Gemopus-4-26B-A4B-it-GGUF`](https://huggingface.co/Jackrong/Gemopus-4-26B-A4B-it-GGUF) | SFT for style consistency / Markdown structure | **Explicitly disclaims tool calling**: "known compatibility issues, not unique to this model" |
| HF blog / TRL fine-tuning recipe | Fine-tune `gemma-4-E2B-it` for tool calling on H100 | Recipe, not a released model |
| Unsloth GGUFs | All sizes, plus fine-tuning guide | No agentic tune |

If you need a robust local agentic small model **today**, the field
still recommends **Qwen3-Coder** family. Gemma 4 is workable for agentic
use with the harness work above, but it's not the path of least
resistance for that workload.

---

## 9. What this means for an existing harness

A concrete checklist if you have a Gemma 4 26B agent in production and
are debugging tool-call reliability:

1. **Check your Ollama version.** Below 0.22.1, upgrade. Below 0.20.6,
   you're missing the first round of Gemma 4 tool-call fixes entirely.
2. **Write the four-condition probe** from § 4 against a known-good
   message state. If only `system + think:False` zeros out
   `tool_calls`, you have #15539, not a model defect.
3. **Set the recipe** from § 4 by loop shape. The safe combo for almost
   all cases is `think: True` + `num_predict: 2048`.
4. **Audit your tool descriptions** for activation verbs. "Call this
   when…" beats "Search for…" by a non-marginal margin per BiasBusters.
5. **Lower temperature to 0.4** if you're at default. This alone took
   one production agent from ~0% to ~80% firing on triggers.
6. **Add a reactive-mode nudge loop**: if `tool_calls=[]` on a turn
   where the user asked something factual, re-prompt with a system
   message asking whether a tool was needed.
7. **If still under-firing**, prototype Probe & Prefill on ~900 labeled
   turns from your own log. This is the lowest-effort high-impact
   technique in the current literature.
8. **Consider Qwen3-Coder for the tool-heavy path** if you're not
   committed to Gemma 4 for other reasons (vision, multilingual,
   licensing).

## Contradictions with prior corpus, flagged

- `GOTCHAS.md` HIGH: "`think: false` Kills Gemma 4 26B in Multi-Turn
  Tool-Calling Loops" — **partially superseded**. The failure is real
  but is parser-side in Ollama (#15539), not model-side. Same fix
  (use `think: True`) but the root cause framing was wrong, and the
  failure also affects e4b, not just 26B.
- `SYNTHESIS.md` "Mandatory Ollama Settings": the single-turn
  "always `think: false`" rule is still correct *for single-turn JSON
  pipelines*. The multi-turn agent guidance needs the recipe from § 4
  applied.
- `GOTCHAS.md` MEDIUM: "Tool Calling Broken in Ollama v0.20.0 Streaming"
  — partially fixed (v0.20.6 + v0.22.1). Re-test on current version
  before assuming streaming tool calls are still broken.
- `GOTCHAS.md` LOW: "`<unused>` Token Infinite Loop (Vulkan backends)"
  — fixed in llama.cpp b8691+.

## Sources

Primary:
- [ollama/ollama#15539](https://github.com/ollama/ollama/issues/15539)
- [ollama/ollama releases](https://github.com/ollama/ollama/releases)
- [ggml-org/llama.cpp#21316](https://github.com/ggml-org/llama.cpp/issues/21316),
  [#21321](https://github.com/ggml-org/llama.cpp/issues/21321),
  [#22080](https://github.com/ggml-org/llama.cpp/issues/22080)
- [vllm-project/vllm#39392](https://github.com/vllm-project/vllm/issues/39392)

Research:
- ["LLM Agents Already Know When to Call Tools" — Probe & Prefill](https://arxiv.org/html/2605.09252v1)
- ["To Call or Not to Call"](https://arxiv.org/html/2605.00737v1)
- [BiasBusters](https://arxiv.org/pdf/2510.00307)
- [NVIDIA When2Call](https://github.com/NVIDIA/When2Call)
- [XGrammar-2 paper](https://arxiv.org/abs/2601.04426),
  [MLC blog](https://blog.mlc.ai/2026/05/04/xgrammar-2-fast-customizable-structured-generation)

Community:
- [Tim Gregg: llama.cpp Gemma 4 thinking-prompt fix](https://complete.tech/blog/llamacpp-gemma4-thinking-prompt-local-fix/)
- [unsloth/gemma-4-26B-A4B-it-GGUF discussion #2 (unused49 root cause)](https://huggingface.co/unsloth/gemma-4-26B-A4B-it-GGUF/discussions/2)
- [google/gemma-4-26B-A4B-it discussion #10 (thinking-mode behavior)](https://huggingface.co/google/gemma-4-26B-A4B-it/discussions/10)