Seth/gemma4-research
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

docs: May 2026 tool-calling update + agentic retrieval landscape

Browse Source 
Two new corpus docs plus targeted GOTCHAS corrections based on
Ollama/llama.cpp/vLLM issue tracker review, late-2025/early-2026
research papers, and convergent empirical evidence from two production
Gemma 4 agents.

New:
- CORPUS_tool_calling_2026-05.md: current state of tool calling.
  Properly characterizes the think flag (parser-side bug, not
  model-side; recipe by loop shape), surfaces ollama/ollama#15539,
  documents the "doesn't fire tools without explicit ask" research
  (Probe & Prefill, BiasBusters, When2Call), XGrammar-2.
- CORPUS_agentic_retrieval.md: post-RAG landscape. Hybrid retrieval
  + cross-encoder rerank as the settled base, CRAG-shaped flows,
  LightRAG/LazyGraphRAG, mem0 vs Letta MemFS, deep-research agents.
  Production-vs-experimental sorting for self-hosted small-model use.

Updates:
- GOTCHAS.md: the think:false rule, Vulkan unused-token loop, and
  Ollama 0.20 streaming bug all marked SUPERSEDED/FIXED/PARTIALLY
  FIXED in-place with pointers to the May update.
- README.md: indexed the two new docs.
- .gitignore: added private/ for bot-specific notes.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Mortdecai
2026-05-25 09:51:09 -04:00
parent 0f82cd71b1
commit 438a96f235
5 changed files with 797 additions and 5 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CORPUS_tool_calling_2026-05.md
+429
View File
@@ -0,0 +1,429 @@
# Gemma 4 Tool-Calling: State of the Art, May 2026
> Six-week update on top of the April 2026 corpus. **Supersedes parts of
> `CORPUS_tool_calling_format.md` and `GOTCHAS.md`** — read this first if
> you're reaching for those documents to debug a tool-calling problem.
>
> Last updated: 2026-05-25. Based on Ollama + llama.cpp + vLLM issue tracker
> review, two academic papers from late 2025, and convergent empirical
> evidence from two independent production agents running `gemma4:26b`.
## TL;DR
1. **The April-2026 rule "`think: false` silently kills `gemma4:26b` in
multi-turn tool-call loops" is wrong as written.** The bug is parser-side
in Ollama, not model-side. It triggers on the **combination** `system
prompt + think:false + tools` — and affects `gemma4:e4b` too, not just
26B MoE. ([ollama/ollama#15539](https://github.com/ollama/ollama/issues/15539))
2. **Ollama has shipped material tool-calling fixes since 0.20.4.** Current
stable is **v0.24.0** (2026-05-14). v0.20.6 explicitly improved Gemma 4
tool calling and parallel-streaming tool calls; v0.22.1 updated the Gemma 4
renderer for thinking and tool calling. If you're below 0.22.1 you're
running on patched-but-incomplete state.
3. **The dominant production failure mode is not "model can't call tools" —
it's "model decides not to call a tool when it should."** This is a
research-characterized problem now, and the best fix in the literature
is **NOT prompt engineering** — it's a lightweight probe on the model's
own hidden states (see § Probe & Prefill).
4. **`think` is not a single boolean rule.** Setting it correctly requires
reasoning about `num_predict` budget, tool-argument length, and prompt
complexity together. The recipe is at the end.
5. **No community fine-tune of Gemma 4 fixes tool-calling reliability** as
of late May 2026. If you need a robust local agentic small model today,
the field still recommends Qwen3-Coder. Gemma 4 is workable but requires
the harness work documented below.
---
## 1. Ollama: what's been fixed, what's still broken
Release cadence April–May 2026: 0.20.4 → 0.20.5 → 0.20.6 → 0.21.0 → 0.21.3 →
0.22.0 → 0.22.1 → 0.23.x → 0.24.0. (0.30.0 is a pre-release reorganizing
llama.cpp consumption and adding MLX on Apple Silicon.)
### Fixed
- **v0.20.6**: "Gemma 4 tool calling ability is improved and updated to use
Google's latest post-launch fixes" + "improved parallel tool calling for
streaming responses." First real fix for the streaming-drops-tool-calls
bug noted in the April corpus.
([release notes](https://github.com/ollama/ollama/releases/tag/v0.20.6))
- **v0.22.1**: "Updated the Gemma 4 renderer for thinking and tool calling
improvements."
([release notes](https://github.com/ollama/ollama/releases/tag/v0.22.1))
- **PR #15467 (closed via merge)**: Added native tool-call parsers for
`gemma4`, `qwen3`, `qwen3-coder`, `cogito`, `deepseek3`, `functiongemma`,
`lfm2`, `ministral`, `olmo3`, `qwen3vl`. Gemma 4 is now in the
"registered parser" list.
### Still open (verify before assuming a bug is gone)
| Issue | Status | What's broken | Workaround |
|-------|--------|---------------|------------|
| [#15539](https://github.com/ollama/ollama/issues/15539) | open, assigned | `system + think:false + tools` produces raw JSON in `content` with trailing `<channel\|>` token and empty `tool_calls`. Affects 26B AND e4b. | Set `think: True`, or remove the system prompt, or move to llama.cpp from source |
| [#15719](https://github.com/ollama/ollama/issues/15719) | open | `gemma4:26b` infinite tool-call loop on 0.20.6/0.20.7 **only when called via LiteLLM proxy** — direct Ollama works, 0.20.5 works | Skip the proxy, or pin Ollama version |
| [#15497](https://github.com/ollama/ollama/issues/15497) | open | OpenAI-compat streaming returns `Function.Index: 0` for every tool call when model has no registered parser | n/a now for Gemma 4 (parser registered); affects custom models |
| [#15315](https://github.com/ollama/ollama/issues/15315) | reopened | Raw `call:tool{...}` text leaks out on e4b when arguments contain JS-style quotes/backticks | Strip / replace special chars in tool args before display |
### Re-reading the April baseline against this
The April corpus said the `think: false` failure was a 26B-MoE-specific
defect — the model emits near-immediate EOS at decision turns. **Issue
#15539's diagnosis suggests this was actually the Ollama parser failing
to extract a tool call from output that, on the wire, contained one.** The
model emitted the call; the framework returned `tool_calls=[]` and an
empty `content` with a stray `<channel|>` token.
If you have a `think: false` 26B harness that's failing, the right next
diagnostic is to **remove the system prompt and re-test** — if it fires
reliably, you've reproduced #15539. The fix is either upgrade Ollama past
0.22.1 and re-verify, or move to llama.cpp built from source with the PRs
in § 2 applied.
---
## 2. llama.cpp: what's been fixed
### Resolved
- **The `<unused49>` / channel-token loop across CUDA, ROCm, Vulkan, SYCL**
was a llama.cpp eval bug, not quantization. Fixed in b8691+. Old quants
work fine on patched builds.
([unsloth/gemma-4-26B-A4B-it-GGUF#2](https://huggingface.co/unsloth/gemma-4-26B-A4B-it-GGUF/discussions/2))
- **Special tokens leaking into tool-call argument values**
(e.g., `[<|"|>light<|"|>]`) — closed via fix in early April.
([ggml-org/llama.cpp#21316](https://github.com/ggml-org/llama.cpp/issues/21316))
- **Canonical "build from source for Gemma 4 tool calling" PR pair**:
[#21326](https://github.com/ggml-org/llama.cpp/pull/21326) (template)
+ [#21343](https://github.com/ggml-org/llama.cpp/pull/21343) (tokenizer).
This combination is cited by multiple writeups as the working baseline.
### Still open or unclear
- **[#22080](https://github.com/ggml-org/llama.cpp/issues/22080)**:
`gemma-4-26B-A4B` "generation stops abruptly mid-output" on multi-GPU
llama-server with `--reasoning-budget` set. No root cause. May be the
same family as Ollama #15539 surfacing in a different runtime.
- **31B + f16/f32 vision projector** still produces `<unused49>` floods on
current builds per one report — the eval-bug fix didn't cover the
multimodal projector path.
- **Inverted thinking-mode guard** in Google's released chat template
(`models/templates/google-gemma-4-31B-it.jinja`): when
`enable_thinking=false`, the upstream template emits a closed empty
thought block; when true, it fails to open `<|channel>thought\n`.
Local-fork fix only as of writing.
([CompleteTech writeup](https://complete.tech/blog/llamacpp-gemma4-thinking-prompt-local-fix/))
`--jinja` is still required. The chat template is the contract.
---
## 3. vLLM: usable, with sharp edges
vLLM ships an official Gemma 4 recipe and a `--tool-call-parser gemma4`
flag. One open caveat:
- **[#39392](https://github.com/vllm-project/vllm/issues/39392)**:
`<pad>` token floods under concurrent requests (2/5 fail at concurrency
5; 0/5 sequential). Reporter suspects shared mutable parser state —
not thread-safe. Workaround is a global serialization lock around the
tool-call path.
vLLM also still suffers the heterogeneous-attention-head Triton fallback
(LOW-severity entry in GOTCHAS.md) — throughput is well below what
similarly-sized models get on the same hardware.
---
## 4. The `think` flag, properly characterized
The April corpus had two contradictory entries: "always `think: false`
for single-turn JSON" and "never `think: false` for multi-turn tool loops
because it kills 26B." Both pointed at the same symptom (empty content,
no usable response) with different prescriptions. Field evidence from two
independent agents running `gemma4:26b` makes the actual interaction
clearer.
### The two failure paths
**Path A — `think: True` + `num_predict` too small.**
The hidden reasoning trace consumes the entire generation budget, leaving
`message.content=""` and `done_reason="length"`. Observed against
`num_predict=1024` with a complex system prompt + tools schema; thinking
trace alone was reaching the cap.
**Path B — `think: False` + (long tool args OR multi-turn agentic loop
OR system prompt present).** Model emits a tool call on the wire that the
parser fails to extract, or emits near-immediate EOS at a decision turn.
Surface symptom: `tool_calls=[]`, empty content, sometimes a stray
`<channel|>` token. Per Ollama #15539 this is parser-side; per the April
bakeoff it also shows up in deep multi-turn loops where each decision
turn is short.
### The recipe
| Loop shape | `think` | `num_predict` | Why |
|------------|---------|---------------|-----|
| Single-turn structured JSON | `False` | ≥ 2048 | No tool-call parsing involved; thinking adds latency without benefit; small budgets are safe |
| Short tool args, ≤ 5-step loop | `False` | ≥ 1024 | Field-verified to fire reliably at temp ≤ 0.5 with explicit Trigger→Action prompt structure |
| Long tool args (image-gen prompts, file payloads, code blocks) | `True` | ≥ 2048 | Field-verified: `think: False` produces empty `tool_calls` on the same context where `think: True` produces a proper call |
| Deep multi-turn agentic (≥ 10 steps) | `True` | ≥ 2048 | Per April bakeoff: `think: False` produces silent EOS at decision turns |
**Safety floor:** if you can afford the latency, `think: True` +
`num_predict: 2048` works for *all* shapes above. The reason to deviate
is one of:
- Latency budget < ~3s per turn (thinking adds 1–2s)
- Predictability is critical (no hidden token generation)
- Schema-constrained output that excludes thinking tokens
**Build a probe.** Both agents that diagnosed this in production wrote a
probe script that called the same Ollama endpoint with each flag setting
against a known-good message state and asserted `tool_calls != []`. If
you're flipping `think`, write the probe first — same-context comparisons
are how the parser-side failure surfaces.
### Reproducing the parser-side diagnosis
To verify whether your harness is hitting Ollama #15539 vs a different
failure path:
```python
# Same model state, four conditions
combos = [
{"system": None, "think": True},
{"system": None, "think": False},
{"system": "...", "think": True},
{"system": "...", "think": False},
]
for c in combos:
response = ollama.chat(model="gemma4:26b", messages=msgs(c), tools=TOOLS,
think=c["think"])
print(c, "→ tool_calls:", len(response["message"]["tool_calls"]))
```
If only `system + think:False` produces zero tool calls, you've reproduced
the parser bug, not the model defect.
---
## 5. The "doesn't fire tools without explicit ask" pattern
This is the dominant production failure mode for small/mid agentic models
as of May 2026 — the model is capable of using tools, calls them when
explicitly told ("look this up", "search for"), and defaults to
answering-from-training otherwise. Now characterized in the literature:
### Quantified
- **["To Call or Not to Call"](https://arxiv.org/html/2605.00737v1)**
measures under/over-calling across model families. **Gemma3-27B over-calls
(462 vs 297 optimal)**; Mistral3.1-24B severely under-calls. Gemma 4 not
directly benched but the family signal is toward over-calling under
reasoning-mode conditions. The "doesn't fire" observation in production
likely reflects `think: false` runs where the model goes straight to
answer without the reasoning step that would surface the tool-call
decision.
- **[BiasBusters](https://arxiv.org/pdf/2510.00307)** quantifies
tool-selection bias driven by description metadata and position. Tools
with **edited** descriptions get **10× more usage** than identical tools
with original descriptions (GPT-4.1, Qwen2.5-7B). The wording of your
tool descriptions is itself tuning firing rate; this is not a marginal
effect.
### What actually moves the needle
**Probe & Prefill** ([arxiv 2605.09252](https://arxiv.org/html/2605.09252v1))
— the most concrete fix in the recent literature, and the one most likely
to apply to a Gemma-4-sized agent.
Core finding: **tool-call decisions are linearly decodable from the
model's pre-generation hidden states with AUROC 0.89–0.96** —
substantially outperforming the model's own verbalized reasoning about
whether to call a tool. The technique:
1. Collect ~900 examples labeled "should call tool" vs "should not."
2. Train a logistic regression on the final-layer hidden state at the
pre-generation position. Trains in seconds on CPU. < 1 ms inference
overhead.
3. At inference, if the probe says "should call," prefill a short
steering sentence ("I'll use a tool for this. ") to bias decoding
toward the tool path.
Demonstrated **48% reduction in unnecessary tool calls with 1.7% accuracy
loss** on Qwen3 (1.7/4/14/32B) and Llama (3.1-8B, 3.3-70B). Not tested
on Gemma 4 — but the technique is model-agnostic and the hidden-state
signal has held across every architecture they tried.
**This is the only technique in the May 2026 literature that beats
careful prompt engineering on this specific problem.** Worth a serious
experiment for any non-trivial Gemma 4 agent.
### Prompt-engineering techniques that help (but won't fully solve)
- **Imperative tool descriptions with activation triggers** beat passive
descriptions. "Call this when the user asks about current events"
fires more reliably than "Search the web for current information."
(BiasBusters' 10× finding is the upper bound; expect smaller effects
in practice.)
- **Trigger → Action contracts in the system prompt** ("If the user
attempts a translation → call `score_vocab`") outperform softer
guidance. Field-verified to take a Gemma 4 26B agent from ~0%
tool-firing at default temperature to ~80% at temp 0.4.
- **Lower temperature** (0.3–0.5) increases firing rate on agentic
loops. Default Ollama temperature (~0.8) biases toward
conversational mimicry of recent chat history.
- **Reactive-mode nudge loop**: if the model emits text without calling
a tool on a turn where tools are available, append a system message
("Was a tool needed? If yes, call it.") and re-prompt. One agent in
the field uses this in its autonomous-research loop but not its
reactive chat loop; the reactive loop has the firing-rate problem
the autonomous loop doesn't.
### Benchmarking your fix
[NVIDIA's When2Call](https://github.com/NVIDIA/When2Call) (NAACL 2025) is
the relevant eval: 1000 MCQs scoring "answer directly / call tool / ask
clarifying / decline." Their finding: **RPO consistently > SFT** for
training models to make this decision well. Useful as the harness even
if you're not training, because it gives you a single number to track
across prompt / parameter changes.
---
## 6. Two-stage tool routing — still alive and well
In-model selection has **not** caught up to two-stage routing for the
"should-call" decision as of May 2026. The router pattern has shifted
from "another LLM" to "small classifier":
- **[ToolCallVerifier](https://huggingface.co/llm-semantic-router/toolcall-verifier)**:
ModernBERT-based 0.1B token classifier. Recall 0.92, precision 0.95,
F1 0.935. Designed for prompt-injection defense but the same pattern
works as a "should-call" gate.
- **vLLM Semantic Router (v0.1 Iris, Jan 2026)**: production framework
for taxonomy-guided routing + confidence-cascade.
- **Probe & Prefill** (above) is the limit case: the "router" is a
logistic regression on the *main model's own* hidden states. No
separate model to host.
**Practical recommendation:** if you're under 5 tools and can't afford
to train a probe, Trigger → Action prompting + temp 0.4 + reactive nudge
loop will get you to ~70–80% firing rate. Beyond that, the probe technique
is the lowest-cost concrete improvement.
---
## 7. Structured output: XGrammar-2 changes the math
**[XGrammar-2](https://blog.mlc.ai/2026/05/04/xgrammar-2-fast-customizable-structured-generation)**
(MLC blog 2026-05-04, [arxiv 2601.04426](https://arxiv.org/abs/2601.04426))
is the headline structured-output development of the year:
- New "Structural Tag" composable JSON protocol uniformly expressing
OpenAI Harmony, tool calling, reasoning channels, and custom output
structures
- Cross-grammar caching with ~50% structure reuse on 50-tool schemas
- Repetition state compression: 534ms → 5.37ms (100×)
- Batching + speculative-decoding support
- **Default structured-generation backend for vLLM, SGLang, TensorRT-LLM,
MLC-LLM** as of March 2026
The per-token overhead is now low enough for production. Smaller models
showed "substantial gains in output accuracy" on BFCL-V3.
**Important distinction:** structured output forces the **format** once
you've decided to call a tool. It does **not** solve "decide whether to
call a tool." For that, see § 5.
Ollama's native `format` schema field (not the broken `format: "json"`)
remains the right Ollama-native option; community reports it's more
reliable than free-form JSON prompting but still leaks for some model
families. No Gemma-4-specific reliability data yet.
---
## 8. New Gemma 4 variants — slim pickings
No community fine-tune released to date materially improves tool-calling
reliability over base Gemma 4 26B-it. Notable releases since April:
| Release | What | Tool-calling relevance |
|---------|------|------------------------|
| [`google/gemma-4-26B-A4B-it-assistant`](https://huggingface.co/google/gemma-4-26B-A4B-it-assistant) | 0.4B drafter for speculative decoding | Generation speed only — not a tool-calling tune |
| `nvidia/Gemma-4-26B-A4B-NVFP4` | NVFP4 quant for Blackwell | Hardware target, not a tune |
| [`Jackrong/Gemopus-4-26B-A4B-it-GGUF`](https://huggingface.co/Jackrong/Gemopus-4-26B-A4B-it-GGUF) | SFT for style consistency / Markdown structure | **Explicitly disclaims tool calling**: "known compatibility issues, not unique to this model" |
| HF blog / TRL fine-tuning recipe | Fine-tune `gemma-4-E2B-it` for tool calling on H100 | Recipe, not a released model |
| Unsloth GGUFs | All sizes, plus fine-tuning guide | No agentic tune |
If you need a robust local agentic small model **today**, the field
still recommends **Qwen3-Coder** family. Gemma 4 is workable for agentic
use with the harness work above, but it's not the path of least
resistance for that workload.
---
## 9. What this means for an existing harness
A concrete checklist if you have a Gemma 4 26B agent in production and
are debugging tool-call reliability:
1. **Check your Ollama version.** Below 0.22.1, upgrade. Below 0.20.6,
you're missing the first round of Gemma 4 tool-call fixes entirely.
2. **Write the four-condition probe** from § 4 against a known-good
message state. If only `system + think:False` zeros out
`tool_calls`, you have #15539, not a model defect.
3. **Set the recipe** from § 4 by loop shape. The safe combo for almost
all cases is `think: True` + `num_predict: 2048`.
4. **Audit your tool descriptions** for activation verbs. "Call this
when…" beats "Search for…" by a non-marginal margin per BiasBusters.
5. **Lower temperature to 0.4** if you're at default. This alone took
one production agent from ~0% to ~80% firing on triggers.
6. **Add a reactive-mode nudge loop**: if `tool_calls=[]` on a turn
where the user asked something factual, re-prompt with a system
message asking whether a tool was needed.
7. **If still under-firing**, prototype Probe & Prefill on ~900 labeled
turns from your own log. This is the lowest-effort high-impact
technique in the current literature.
8. **Consider Qwen3-Coder for the tool-heavy path** if you're not
committed to Gemma 4 for other reasons (vision, multilingual,
licensing).
## Contradictions with prior corpus, flagged
- `GOTCHAS.md` HIGH: "`think: false` Kills Gemma 4 26B in Multi-Turn
Tool-Calling Loops" — **partially superseded**. The failure is real
but is parser-side in Ollama (#15539), not model-side. Same fix
(use `think: True`) but the root cause framing was wrong, and the
failure also affects e4b, not just 26B.
- `SYNTHESIS.md` "Mandatory Ollama Settings": the single-turn
"always `think: false`" rule is still correct *for single-turn JSON
pipelines*. The multi-turn agent guidance needs the recipe from § 4
applied.
- `GOTCHAS.md` MEDIUM: "Tool Calling Broken in Ollama v0.20.0 Streaming"
— partially fixed (v0.20.6 + v0.22.1). Re-test on current version
before assuming streaming tool calls are still broken.
- `GOTCHAS.md` LOW: "`<unused>` Token Infinite Loop (Vulkan backends)"
— fixed in llama.cpp b8691+.
## Sources
Primary:
- [ollama/ollama#15539](https://github.com/ollama/ollama/issues/15539)
- [ollama/ollama releases](https://github.com/ollama/ollama/releases)
- [ggml-org/llama.cpp#21316](https://github.com/ggml-org/llama.cpp/issues/21316),
[#21321](https://github.com/ggml-org/llama.cpp/issues/21321),
[#22080](https://github.com/ggml-org/llama.cpp/issues/22080)
- [vllm-project/vllm#39392](https://github.com/vllm-project/vllm/issues/39392)
Research:
- ["LLM Agents Already Know When to Call Tools" — Probe & Prefill](https://arxiv.org/html/2605.09252v1)
- ["To Call or Not to Call"](https://arxiv.org/html/2605.00737v1)
- [BiasBusters](https://arxiv.org/pdf/2510.00307)
- [NVIDIA When2Call](https://github.com/NVIDIA/When2Call)
- [XGrammar-2 paper](https://arxiv.org/abs/2601.04426),
[MLC blog](https://blog.mlc.ai/2026/05/04/xgrammar-2-fast-customizable-structured-generation)
Community:
- [Tim Gregg: llama.cpp Gemma 4 thinking-prompt fix](https://complete.tech/blog/llamacpp-gemma4-thinking-prompt-local-fix/)
- [unsloth/gemma-4-26B-A4B-it-GGUF discussion #2 (unused49 root cause)](https://huggingface.co/unsloth/gemma-4-26B-A4B-it-GGUF/discussions/2)
- [google/gemma-4-26B-A4B-it discussion #10 (thinking-mode behavior)](https://huggingface.co/google/gemma-4-26B-A4B-it/discussions/10)