Seth/gemma4-research
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

docs: May 2026 tool-calling update + agentic retrieval landscape

Browse Source 
Two new corpus docs plus targeted GOTCHAS corrections based on
Ollama/llama.cpp/vLLM issue tracker review, late-2025/early-2026
research papers, and convergent empirical evidence from two production
Gemma 4 agents.

New:
- CORPUS_tool_calling_2026-05.md: current state of tool calling.
  Properly characterizes the think flag (parser-side bug, not
  model-side; recipe by loop shape), surfaces ollama/ollama#15539,
  documents the "doesn't fire tools without explicit ask" research
  (Probe & Prefill, BiasBusters, When2Call), XGrammar-2.
- CORPUS_agentic_retrieval.md: post-RAG landscape. Hybrid retrieval
  + cross-encoder rerank as the settled base, CRAG-shaped flows,
  LightRAG/LazyGraphRAG, mem0 vs Letta MemFS, deep-research agents.
  Production-vs-experimental sorting for self-hosted small-model use.

Updates:
- GOTCHAS.md: the think:false rule, Vulkan unused-token loop, and
  Ollama 0.20 streaming bug all marked SUPERSEDED/FIXED/PARTIALLY
  FIXED in-place with pointers to the May update.
- README.md: indexed the two new docs.
- .gitignore: added private/ for bot-specific notes.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Mortdecai
2026-05-25 09:51:09 -04:00
parent 0f82cd71b1
commit 438a96f235
5 changed files with 797 additions and 5 deletions
Download Patch File Download Diff File Expand all files Collapse all files
GOTCHAS.md
+55 -5
View File
@@ -2,6 +2,13 @@
> Derived from Seth's production implementations (Simon, AI_Visualizer)
> and community reports. These are hard-won lessons.
>
> **2026-05-25 corrections:** several entries below were corrected by
> field evidence from two independent agents and by Ollama/llama.cpp
> issue-tracker review. See `CORPUS_tool_calling_2026-05.md` for the
> consolidated current state. The "`think: false` kills 26B" rule and
> the `<unused>` Vulkan-loop entry both need re-reading against that
> update — they are marked **SUPERSEDED** in place below.
## CRITICAL: Thinking Mode Eats Context (single-turn pipelines only)
@@ -75,10 +82,32 @@ Ollama defaults `num_predict` to 128 tokens. Almost any useful Gemma 4 output ex
**Fix:** Always set `num_predict` explicitly. Minimum recommended: 512. For JSON output: 2048+.
## HIGH: `think: false` Kills Gemma 4 26B in Multi-Turn Tool-Calling Loops
## HIGH: `think: false` Kills Gemma 4 26B in Multi-Turn Tool-Calling Loops [SUPERSEDED 2026-05-25]
**Severity: HIGH — silent agent-loop failure. Setting is what the old guidance said to do.**
> **SUPERSEDED 2026-05-25:** the symptom is real but the diagnosis was
> wrong. Per [ollama/ollama#15539](https://github.com/ollama/ollama/issues/15539),
> the failure is an **Ollama parser bug** triggered by the combination
> `system prompt + think:false + tools` — the model emits a valid tool
> call on the wire, the framework returns `tool_calls=[]` with an empty
> `content` and a stray `<channel|>` token. **The bug affects `gemma4:e4b`
> too, not just 26B MoE.** Field evidence from a production agent at
> `num_predict=1024` confirms `think: false` works fine for short tool
> arguments and shallow loops — the original bakeoff hit the deep-loop
> case where the parser fails.
>
> **Replacement rule:** see `CORPUS_tool_calling_2026-05.md` § 4 for the
> proper recipe by loop shape. Safe default for all cases:
> `think: True` + `num_predict ≥ 2048`. Diagnostic for whether you're
> hitting this bug: run a four-condition probe (system × think) at the
> same model state — if only `system + think:False` zeros tool_calls,
> you've reproduced #15539.
>
> The April-2026 content below is preserved for the historical record but
> should not be applied without first reading the May update.
Reproduced on 2026-04-18 against `gemma4:26b` via Ollama 0.20.4 on a 3090 Ti
(steel141). Contradicts the older "always think:false" guidance (see § "Thinking
Mode Eats Context" below — now scoped to single-turn pipelines only).
@@ -178,9 +207,9 @@ Community-reported: Flash Attention causes Gemma 4 31B Dense to hang indefinitel
**Fix:** Use 26B for long prompts, or disable Flash Attention if running 31B on affected hardware.
## MEDIUM: Tool Calling Broken in Ollama v0.20.0 Streaming
## MEDIUM: Tool Calling Broken in Ollama v0.20.0 Streaming [PARTIALLY FIXED]
**Severity: MEDIUM — version-specific**
**Severity: MEDIUM — version-specific. PARTIALLY FIXED in v0.20.6 + v0.22.1.**
As of early April 2026, Gemma 4 tool calling has issues in Ollama v0.20.0: the tool call parser fails and streaming drops tool calls entirely. Community reports include format mismatches and continuous loops in llama.cpp / LM Studio.
@@ -188,6 +217,17 @@ As of early April 2026, Gemma 4 tool calling has issues in Ollama v0.20.0: the t
**Fix:** Use non-streaming for tool calls (Simon does this). Test tool calling thoroughly when upgrading Ollama versions. Seth's implementations work reliably with non-streaming tool calls.
> **2026-05-25 update:** Ollama [v0.20.6](https://github.com/ollama/ollama/releases/tag/v0.20.6)
> explicitly improved Gemma 4 tool-calling and added "improved parallel
> tool calling for streaming responses." [v0.22.1](https://github.com/ollama/ollama/releases/tag/v0.22.1)
> updated the Gemma 4 renderer for thinking + tool-calling improvements.
> Current stable is **v0.24.0** (2026-05-14). Re-test streaming on current
> versions before assuming the original bug persists. **Open bugs**
> remain — see `CORPUS_tool_calling_2026-05.md` § 1 for the current list,
> especially [#15539](https://github.com/ollama/ollama/issues/15539)
> (`system + think:false + tools` returns empty tool_calls) which is the
> most likely failure mode you'll hit on a recent Ollama.
## MEDIUM: VRAM-Hungry for Context
**Severity: MEDIUM — affects hardware planning**
@@ -222,14 +262,24 @@ Heterogeneous attention head dimensions in Gemma 4 force vLLM to fall back to a
**Fix:** Use Ollama instead of vLLM for now, or wait for the fix.
## LOW: `<unused>` Token Infinite Loop (Vulkan backends)
## LOW: `<unused>` Token Infinite Loop (Vulkan backends) [FIXED]
**Severity: LOW — Vulkan-specific**
**Severity: LOW — Vulkan-specific. FIXED in llama.cpp b8691+.**
Gemma 4 can generate `<unused>` or `<unused24>` tokens in an infinite loop on Vulkan backends in llama.cpp.
**Source:** [ggml-org/llama.cpp#21516](https://github.com/ggml-org/llama.cpp/issues/21516)
> **2026-05-25:** Per the
> [unsloth/gemma-4-26B-A4B-it-GGUF discussion thread](https://huggingface.co/unsloth/gemma-4-26B-A4B-it-GGUF/discussions/2),
> the `<unused49>` / channel-token flood across CUDA / ROCm / Vulkan / SYCL
> was a llama.cpp eval bug, **not** a quantization problem. Fixed in
> b8691+. Old quants work fine on patched builds.
>
> Caveat: 31B + f16/f32 vision projector still produces `<unused49>`
> floods on current builds per one report — the eval-bug fix didn't
> cover the multimodal projector path.
## MEDIUM: `google/gemma_pytorch` Abandoned for Gemma 4
**Severity: MEDIUM — wastes time on a dead-end path**