diff --git a/.gitignore b/.gitignore index 2a2c38d..4c8f6ea 100644 --- a/.gitignore +++ b/.gitignore @@ -13,3 +13,6 @@ scripts/bakeoff/runs*/*/work/ # Editor / OS .DS_Store *.swp + +# Private — bot-specific findings, recommendations tied to non-public projects +private/ diff --git a/CORPUS_agentic_retrieval.md b/CORPUS_agentic_retrieval.md new file mode 100644 index 0000000..4579509 --- /dev/null +++ b/CORPUS_agentic_retrieval.md @@ -0,0 +1,308 @@ +# Agentic Information Retrieval Beyond RAG, May 2026 + +> A view of where retrieval and memory architectures are converging for +> agentic applications, with explicit production-vs-experimental sorting +> for self-hosted small-model deployments (Gemma 4 26B–31B, Qwen3 8–14B). +> +> Last updated: 2026-05-25. Cuts off long-context-as-RAG-replacement, +> graph retrieval, agent memory, and deep-research-agent patterns. + +## Headlines + +1. **The 2023-era named patterns (Self-RAG, CRAG, FLARE, MemGPT) survived + as design templates but their original codebases are mostly stale.** + The 2026 production stack reimplements them as graph nodes inside + LangGraph or LlamaIndex Workflows. Use the patterns; don't try to use + the original repos. +2. **Long context did not replace RAG.** Multi-needle retrieval at 1M + tokens regressed in some frontier models in 2026; even where it works, + the cost/latency math means RAG-then-stuff stays the winning shape. + The "just dump the corpus" approach is now a niche choice for + single-document deep reasoning, not corpus retrieval. +3. **GraphRAG got cost-viable.** Microsoft's + [LazyGraphRAG](https://www.microsoft.com/en-us/research/blog/lazygraphrag-setting-a-new-standard-for-quality-and-cost/) + matches or beats full GraphRAG quality at indexing cost equal to vector + RAG and query cost 0.1% of full GraphRAG. GA targeting Q1–Q2 2026 in + the MSR repo. For SMB corpora today, [LightRAG](https://github.com/hkuds/lightrag) + is the production option. +4. **Memory architectures split into two camps**: managed CRUD layer + (mem0, Zep) vs. agent-edited state (Letta with Context Repositories + "MemFS", A-MEM). For small open models, lean toward managed — Gemma 4 + will sometimes drop tool calls and you'll lose memory silently if the + model owns it. +5. **The settled retrieval base layer is hybrid BM25 + dense + RRF + fusion + cross-encoder rerank.** "Dense embeddings alone" has been + retired in serious systems. The lexical signal is load-bearing for + error codes, identifiers, and technical corpora that embeddings + underperform on. + +--- + +## 1. Agentic RAG — patterns, not libraries + +The original papers all date from 2023. In 2026 they exist as patterns +absorbed into agent frameworks. + +| Pattern | Origin | What survived | What to use | +|---------|--------|---------------|-------------| +| **Self-RAG** | Asai et al., ICLR 2024 | Reflection-token idea (model decides per-step whether to retrieve, critique chunks) | Re-implement as graph nodes; the trained 7B/13B models aren't being updated | +| **CRAG (Corrective RAG)** | Yan et al., 2024 | retrieve → evaluate → correct → generate | LangGraph cookbook implementation is now the canonical artifact | +| **FLARE** | Jiang et al., 2023 | Uncertainty-triggered retrieval (low-confidence span ⇒ re-retrieve) | Pattern survives, original repo stale | +| **PRISM / RELOOP** | arxiv 2510.14278, 2510.20505 (late 2025) | New, multi-hop with precision-focused control flow | Experimental — track if multi-hop precision is your bottleneck | + +**Frameworks ranked by fit for branching agentic RAG flows:** + +- **LangGraph** (graph-native, best for CRAG/Self-RAG-shaped branching) +- **LlamaIndex Workflows** (event-driven, similar story) +- **Haystack** (more structured pipelines, weaker on dynamic branching) + +GitHub star order, January 2026: +LangChain 125K · Dify 114K · RAGFlow 70K · LlamaIndex 46.5K. +([source](https://florinelchis.medium.com/top-10-rag-frameworks-on-github-by-stars-january-2026-e6edff1e0d91)) + +**Convergence:** yes, on the pattern — agent decides when to retrieve, +evaluates retrieved chunks, optionally re-queries. **No convergence** on +which framework owns it. + +--- + +## 2. GraphRAG: the cost story + +Microsoft's original GraphRAG had a ~$33K indexing bill on large corpora, +making it a non-starter outside well-funded enterprises. 2026 changed +that. + +| Project | Maturity | License | When to use | +|---------|----------|---------|-------------| +| [Microsoft GraphRAG](https://github.com/microsoft/graphrag) | Production but expensive | MIT | Global-question answering over stable corpora when budget is unconstrained | +| [LazyGraphRAG](https://www.microsoft.com/en-us/research/blog/lazygraphrag-setting-a-new-standard-for-quality-and-cost/) | Beta, GA Q1–Q2 2026 | MIT | The new default — same quality, 0.1% query cost | +| [LightRAG](https://github.com/hkuds/lightrag) (EMNLP 2025) | Production | MIT | SMB corpora today, Docker-deployable, PG+pgvector+AGE one-DB path | +| [nano-graphrag](https://github.com/gusye1234/nano-graphrag), [fast-graphrag](https://github.com/circlemind-ai/fast-graphrag) | Experimental | MIT | Hackable baselines, sub-1M-doc cases | + +**2026 consensus on graph vs. vector** (per +[GraphRAG-Bench, ICLR'26](https://github.com/GraphRAG-Bench/GraphRAG-Benchmark)): +graph wins when (a) coherent domain corpus, (b) questions require +synthesizing across many documents, (c) indexing pass is affordable. +Vector wins on point-fact lookup, noisy/heterogeneous corpora, cost, +latency. + +**Recommendation:** for a self-hosted small-model setup with a coherent +domain corpus, LightRAG is the production try-this. Skip full Microsoft +GraphRAG (cost). Watch for LazyGraphRAG GA and swap when shipped. + +--- + +## 3. Long context did not replace RAG + +The 2026 reality killed the "just stuff the corpus" dream. + +- **Claude Opus 4.7** regressed on multi-needle 1M-context retrieval — + 32.2% on MRCR v2 8-needle at 1M vs. 78.3% for Opus 4.6. + ([source](https://blog.wentuo.ai/en/claude-opus-4-7-long-context-regression-en.html)) +- Multi-needle leaderboard at 1M tokens: Gemini 3 leads at 89%, + GPT-5.5 at 74%, Opus 4.7 at 56%. + ([source](https://www.digitalapplied.com/blog/gpt-5-5-vs-claude-opus-4-7-frontier-comparison)) +- **RAGAS 2.0 (April 2026) study** found faithfulness dropped up to **40%** + when context windows were overloaded with irrelevant filler. + ([source](https://open-techstack.com/blog/rag-vs-long-context-2026/)) + +**Effective context for multi-needle workloads is 200–400K** for current +frontier models even when the marketed window is 1M+. Above that, attention +dilutes, "lost in the middle" kicks in, latency explodes, cost is linear. + +**Gemma 4 specifically: 128K context.** Plenty for the retrieve-then-stuff +pattern, but cap effective working context at ~64K to stay above the +lost-in-the-middle floor for 8–31B models. Not a corpus replacement. + +**The winning shape:** RAG retrieves the top 50–200K most relevant tokens +→ long-context model reasons carefully. "Just dump everything" is the +niche choice for single long-document analysis (legal contract, research +paper). + +--- + +## 4. Memory architectures + +Real movement in 2025–2026. Five-way taxonomy +([SurePrompts](https://sureprompts.com/blog/agent-memory-architectures-compared-2026)): +provider-managed, self-managing (Letta), CRUD memory layer (mem0), vector +RAG, custom in-app schema. + +| Project | Camp | Maturity | License | Best for | +|---------|------|----------|---------|----------| +| [Letta](https://github.com/letta-ai/letta) (formerly MemGPT) | Agent-edited state. **Feb 2026: Context Repositories ("MemFS")** — memory projected into git-backed files operated on via bash/computer-use tools | Production | Apache 2.0 | Long-horizon coherence as the product, strong models only | +| [mem0](https://mem0.ai/research) | CRUD memory layer, vector + optional graph, three-level hierarchy | Production | Apache 2.0 | "Remember the user across sessions" consumer-app cases | +| Zep | Graph-based temporal memory, claims 75.14% on LoCoMo (contested vs mem0) | Production | Commercial + community OSS | Temporal recall + graph relationships | +| [A-MEM](https://github.com/agiresearch/a-mem) (arxiv 2502.12110) | Zettelkasten-inspired — adding new memories triggers updates to existing ones, evolving graph | Experimental | Research | If you want to play with self-organizing memory graphs | +| Hindsight, Memvid, Supermemory, MemPalace | Various | Experimental | Various | Watch, don't commit | + +**For Gemma-4-sized models specifically:** lean toward mem0-style managed +memory. Letta's self-edit pattern needs a model that reliably calls the +memory-save tool every time it should — Gemma 4 won't. The benchmarking +fight between mem0 and Zep on LoCoMo is contested; pick on integration +fit, not on the leaderboard. + +--- + +## 5. Retrieval-augmented planning and multi-hop + +The 2023-era patterns (ReAct, IRCoT, Self-Ask, DRAGIN) remain the +conceptual foundation. Known weaknesses: error propagation, noisy evidence +sets, fixed step budgets. + +**Practical pattern that converged in 2026:** +``` +ReAct loop + cross-encoder reranker on every retrieval + + explicit "do I have enough?" verification step before answering +``` + +The verification step is the real change from 2023. Agents are expected +to check their evidence rather than just synthesize. + +**Recent academic:** +- [PRISM (arxiv 2510.14278)](https://arxiv.org/pdf/2510.14278) and + [RELOOP (arxiv 2510.20505)](https://arxiv.org/pdf/2510.20505) — late + 2025 frameworks attempting to fix the precision-recall imbalance of + IRCoT-style approaches. Experimental. +- [Four-axis design framework survey (arxiv 2601.00536)](https://arxiv.org/html/2601.00536v1) + — useful 2026 organization of the design space. + +--- + +## 6. Deep-research agents + +Heavy activity. OpenAI Deep Research (early 2025) set the template; +open-source caught up fast. + +| Project | Maturity | License | Notes | +|---------|----------|---------|-------| +| OpenAI / Google Deep Research | Closed | — | Reference target. OpenAI ~67% GAIA validation | +| [smolagents/open_deep_research](https://github.com/huggingface/smolagents/tree/main/examples/open_deep_research) | Production code, experimental quality with small models | Apache 2.0 | 55% GAIA validation with GPT-4o. CodeAgent uses ~30% fewer steps than ToolCallingAgent | +| [gpt-researcher](https://github.com/assafelovic/gpt-researcher) | Production | Apache 2.0 | Most mature open option. Planner + search + reader + report writer with citations | +| [LangChain open_deep_research](https://github.com/langchain-ai/open_deep_research) | Production | MIT | Tavily search default, MCP support, model-agnostic | +| Perplexity Sonar API | Production, closed | — | 37% citation hallucination rate (vs ChatGPT Search 67%, Grok 3 94%) per Columbia Journalism Review audit. Best closed-source citation grounding | + +**For Gemma 4 specifically:** 7B–13B class models work in deep-research +frameworks but quality drops noticeably vs. GPT-4o-class. **The 31B +variant is probably the sweet spot for self-hosted deep research; +smaller variants struggle with the planning step.** + +--- + +## 7. Hybrid / late-interaction retrievers — alive and well + +"Dense + reranker won" is **not** the 2026 consensus. + +| Approach | Maturity | License | When | +|----------|----------|---------|------| +| **ColBERT v2** late-interaction, per-token embeddings | Production | Apache 2.0 | Retrieval-heavy reference architecture | +| [**ColPali**](https://arxiv.org/html/2407.01449v5) — late-interaction over document images via VLMs, OCR-free | Beta → production transition | Apache 2.0 | Visually rich documents (tables, charts, forms). Storage cost is the gotcha (multi-vector per page). Zilliz/Milvus and Weaviate ship integrations | +| **SPLADE** — learned sparse, lexical + semantic | Production | Apache 2.0 | Hybrid pipelines. Lexical signal load-bearing for technical corpora | +| [**BGE-M3**](https://bge-model.com/bge/bge_m3.html) — dense + sparse + multi-vector in one model | Production | MIT | Multilingual default | +| [**Qwen3-Embedding**](https://github.com/QwenLM/Qwen3-Embedding) (0.6B/4B/8B) — 8B topped MTEB multilingual at 70.58 mid-2025 | Production | Apache 2.0 | Strong open default, especially multilingual | + +**Settled production pattern** +([source](https://tianpan.co/blog/2026-04-12-hybrid-search-production-bm25-dense-embeddings)): + +``` +BM25 + dense ANN (parallel) → RRF fusion → top-100 candidates + → cross-encoder rerank → top-5 → LLM +``` + +BM25 stubbornly wins on literal matches (error codes, product SKUs, +identifiers) — embeddings underperform on those. "Embeddings alone" has +been mostly retired in serious systems. + +--- + +## 8. Reranking and verification + +Cross-encoder reranking is table stakes. Choice of model: + +| Reranker | License | Notes | +|----------|---------|-------| +| [BGE-reranker-v2-m3](https://huggingface.co/BAAI/bge-reranker-v2-m3) | MIT | Multilingual, self-hostable, price/performance default | +| Cohere Rerank v3.5 | Closed/paid | 4096-token context, JSON / semi-structured. "Just works" choice | +| [Jina Reranker v3](https://jina.ai/models/jina-reranker-v3/) | Apache 2.0 | 0.6B listwise (query + all candidates in one window). 81.33% Hit@1 at 188ms | +| Qwen3 Reranker (0.6B/4B/8B) | Apache 2.0 | Competitive with BGE/Jina | +| Voyage rerank-2.5, mixedbread mxbai, nemotron reranker | Various | Also in the mix. Nemotron at 83% Hit@1 / 243ms currently the accuracy leader per AIMultiple benchmark | + +**Verification patterns (genuinely new in 2026):** +- Explicit "is this passage relevant?" LLM call after rerank +- Citation-supported answers — every claim must point to a retrieved + passage +- Groundedness scores via [Ragas](https://github.com/explodinggradients/ragas) + +**Ragas production thresholds**: faithfulness ≥ 0.9, answer relevancy +≥ 0.85, context precision ≥ 0.8. + +--- + +## 9. Production-vs-experimental quick reference + +| Thing | Maturity | License | When to use | +|-------|----------|---------|-------------| +| Hybrid BM25 + dense + RRF + cross-encoder rerank | **Production**, settled | varies | Default starting point for any new RAG | +| BGE-reranker-v2-m3 / Jina v3 / Qwen3-Reranker | **Production** | MIT/Apache | The reranker step above | +| LangGraph / LlamaIndex Workflows | **Production** | MIT | Implementing CRAG / Self-RAG–shaped flows | +| gpt-researcher / smolagents open_deep_research | **Production** (open) | Apache/MIT | Deep-research agents | +| LightRAG | **Production** for SMB corpora | MIT | Graph-aware retrieval without MSR cost | +| LazyGraphRAG | **Beta** (Q1–Q2 2026 GA) | MIT | Replacing full GraphRAG when shipped | +| mem0 | **Production** | Apache 2.0 | "Remember the user" memory | +| Letta + Context Repositories (MemFS) | **Beta–Production** | Apache 2.0 | Long-horizon agent state, strong models only | +| ColPali | **Beta–production** | Apache 2.0 | Visually rich documents, accept storage cost | +| Long context as RAG-replacement | **Niche** | n/a | Single long document, not corpus | +| Self-RAG original models, FLARE original repo | **Paper-only** | research | Read the paper, re-implement the pattern | +| A-MEM | **Experimental** | research | Self-organizing memory graph experiments | +| PRISM / RELOOP | **Experimental** | research | Multi-hop precision experiments | + +**Genuine convergence:** +- Hybrid retrieval + cross-encoder rerank is the universal base. +- CRAG-shaped self-correction is the dominant agentic RAG pattern. +- Long context complements RAG; doesn't replace it. +- Citations / grounded answers are expected. + +**Still no consensus on:** +- Which agent framework wins (LangGraph vs LlamaIndex Workflows vs + smolagents vs roll-your-own). +- Memory: managed CRUD layer vs agent-edited state. +- When graph beats vector for a given corpus (domain-dependent). +- Whether tool-call agents or code-writing agents are the better + substrate (smolagents data favors code; production deployments split). +- Memory benchmarking — LoCoMo numbers actively contested. + +--- + +## 10. Picking a stack for a Gemma 4 agent + +For a self-hosted Gemma 4 26B/31B agent that already does basic +tool-calling + embedding RAG and wants to step up: + +1. **Base retrieval**: hybrid BM25 + dense (Qwen3-Embedding-4B or + BGE-M3) with RRF fusion, then BGE-reranker-v2-m3 cross-encoder. + Non-experimental but the foundation everything else assumes. +2. **Agentic layer**: CRAG-shaped flow in LangGraph — evaluate retrieved + chunks, fall back to web search if score is low, verify before + answering. Skip Self-RAG-the-models (won't work with Gemma); use + the pattern. +3. **Memory**: mem0 for user-facing recall. Skip Letta unless the agent + specifically needs long-horizon autonomous behavior — Gemma 4 will + sometimes drop tool calls and you'll lose memory silently if the + model owns it. +4. **Web search**: gpt-researcher or smolagents/open_deep_research, + with Gemma 4 31B doing the planning/synthesis. Expect 30–40% lower + quality than GPT-4o-class; budget for it. +5. **Graph experiment**: LightRAG on whichever surface has a coherent + domain corpus. Skip full MSR GraphRAG (cost). Watch for LazyGraphRAG + GA and swap. +6. **Long context**: don't lean on it. 128K Gemma 4 context is for + "retrieve a lot then reason," not "skip retrieval." Cap effective + working context at ~64K. +7. **Worth a one-off experiment**: ColPali on any visual-document + corpus (genuinely different OCR-free path), A-MEM for memory if + mem0 feels too static, PRISM-style multi-hop if you have a benchmark + question your CRAG flow can't handle. + +**Don't spend time on**: original 2023 implementations (Self-RAG models, +FLARE repo, MemGPT pre-Letta). The ideas survived; the codebases did +not. diff --git a/CORPUS_tool_calling_2026-05.md b/CORPUS_tool_calling_2026-05.md new file mode 100644 index 0000000..f04b3bc --- /dev/null +++ b/CORPUS_tool_calling_2026-05.md @@ -0,0 +1,429 @@ +# Gemma 4 Tool-Calling: State of the Art, May 2026 + +> Six-week update on top of the April 2026 corpus. **Supersedes parts of +> `CORPUS_tool_calling_format.md` and `GOTCHAS.md`** — read this first if +> you're reaching for those documents to debug a tool-calling problem. +> +> Last updated: 2026-05-25. Based on Ollama + llama.cpp + vLLM issue tracker +> review, two academic papers from late 2025, and convergent empirical +> evidence from two independent production agents running `gemma4:26b`. + +## TL;DR + +1. **The April-2026 rule "`think: false` silently kills `gemma4:26b` in + multi-turn tool-call loops" is wrong as written.** The bug is parser-side + in Ollama, not model-side. It triggers on the **combination** `system + prompt + think:false + tools` — and affects `gemma4:e4b` too, not just + 26B MoE. ([ollama/ollama#15539](https://github.com/ollama/ollama/issues/15539)) +2. **Ollama has shipped material tool-calling fixes since 0.20.4.** Current + stable is **v0.24.0** (2026-05-14). v0.20.6 explicitly improved Gemma 4 + tool calling and parallel-streaming tool calls; v0.22.1 updated the Gemma 4 + renderer for thinking and tool calling. If you're below 0.22.1 you're + running on patched-but-incomplete state. +3. **The dominant production failure mode is not "model can't call tools" — + it's "model decides not to call a tool when it should."** This is a + research-characterized problem now, and the best fix in the literature + is **NOT prompt engineering** — it's a lightweight probe on the model's + own hidden states (see § Probe & Prefill). +4. **`think` is not a single boolean rule.** Setting it correctly requires + reasoning about `num_predict` budget, tool-argument length, and prompt + complexity together. The recipe is at the end. +5. **No community fine-tune of Gemma 4 fixes tool-calling reliability** as + of late May 2026. If you need a robust local agentic small model today, + the field still recommends Qwen3-Coder. Gemma 4 is workable but requires + the harness work documented below. + +--- + +## 1. Ollama: what's been fixed, what's still broken + +Release cadence April–May 2026: 0.20.4 → 0.20.5 → 0.20.6 → 0.21.0 → 0.21.3 → +0.22.0 → 0.22.1 → 0.23.x → 0.24.0. (0.30.0 is a pre-release reorganizing +llama.cpp consumption and adding MLX on Apple Silicon.) + +### Fixed + +- **v0.20.6**: "Gemma 4 tool calling ability is improved and updated to use + Google's latest post-launch fixes" + "improved parallel tool calling for + streaming responses." First real fix for the streaming-drops-tool-calls + bug noted in the April corpus. + ([release notes](https://github.com/ollama/ollama/releases/tag/v0.20.6)) +- **v0.22.1**: "Updated the Gemma 4 renderer for thinking and tool calling + improvements." + ([release notes](https://github.com/ollama/ollama/releases/tag/v0.22.1)) +- **PR #15467 (closed via merge)**: Added native tool-call parsers for + `gemma4`, `qwen3`, `qwen3-coder`, `cogito`, `deepseek3`, `functiongemma`, + `lfm2`, `ministral`, `olmo3`, `qwen3vl`. Gemma 4 is now in the + "registered parser" list. + +### Still open (verify before assuming a bug is gone) + +| Issue | Status | What's broken | Workaround | +|-------|--------|---------------|------------| +| [#15539](https://github.com/ollama/ollama/issues/15539) | open, assigned | `system + think:false + tools` produces raw JSON in `content` with trailing `` token and empty `tool_calls`. Affects 26B AND e4b. | Set `think: True`, or remove the system prompt, or move to llama.cpp from source | +| [#15719](https://github.com/ollama/ollama/issues/15719) | open | `gemma4:26b` infinite tool-call loop on 0.20.6/0.20.7 **only when called via LiteLLM proxy** — direct Ollama works, 0.20.5 works | Skip the proxy, or pin Ollama version | +| [#15497](https://github.com/ollama/ollama/issues/15497) | open | OpenAI-compat streaming returns `Function.Index: 0` for every tool call when model has no registered parser | n/a now for Gemma 4 (parser registered); affects custom models | +| [#15315](https://github.com/ollama/ollama/issues/15315) | reopened | Raw `call:tool{...}` text leaks out on e4b when arguments contain JS-style quotes/backticks | Strip / replace special chars in tool args before display | + +### Re-reading the April baseline against this + +The April corpus said the `think: false` failure was a 26B-MoE-specific +defect — the model emits near-immediate EOS at decision turns. **Issue +#15539's diagnosis suggests this was actually the Ollama parser failing +to extract a tool call from output that, on the wire, contained one.** The +model emitted the call; the framework returned `tool_calls=[]` and an +empty `content` with a stray `` token. + +If you have a `think: false` 26B harness that's failing, the right next +diagnostic is to **remove the system prompt and re-test** — if it fires +reliably, you've reproduced #15539. The fix is either upgrade Ollama past +0.22.1 and re-verify, or move to llama.cpp built from source with the PRs +in § 2 applied. + +--- + +## 2. llama.cpp: what's been fixed + +### Resolved + +- **The `` / channel-token loop across CUDA, ROCm, Vulkan, SYCL** + was a llama.cpp eval bug, not quantization. Fixed in b8691+. Old quants + work fine on patched builds. + ([unsloth/gemma-4-26B-A4B-it-GGUF#2](https://huggingface.co/unsloth/gemma-4-26B-A4B-it-GGUF/discussions/2)) +- **Special tokens leaking into tool-call argument values** + (e.g., `[<|"|>light<|"|>]`) — closed via fix in early April. + ([ggml-org/llama.cpp#21316](https://github.com/ggml-org/llama.cpp/issues/21316)) +- **Canonical "build from source for Gemma 4 tool calling" PR pair**: + [#21326](https://github.com/ggml-org/llama.cpp/pull/21326) (template) + + [#21343](https://github.com/ggml-org/llama.cpp/pull/21343) (tokenizer). + This combination is cited by multiple writeups as the working baseline. + +### Still open or unclear + +- **[#22080](https://github.com/ggml-org/llama.cpp/issues/22080)**: + `gemma-4-26B-A4B` "generation stops abruptly mid-output" on multi-GPU + llama-server with `--reasoning-budget` set. No root cause. May be the + same family as Ollama #15539 surfacing in a different runtime. +- **31B + f16/f32 vision projector** still produces `` floods on + current builds per one report — the eval-bug fix didn't cover the + multimodal projector path. +- **Inverted thinking-mode guard** in Google's released chat template + (`models/templates/google-gemma-4-31B-it.jinja`): when + `enable_thinking=false`, the upstream template emits a closed empty + thought block; when true, it fails to open `<|channel>thought\n`. + Local-fork fix only as of writing. + ([CompleteTech writeup](https://complete.tech/blog/llamacpp-gemma4-thinking-prompt-local-fix/)) + +`--jinja` is still required. The chat template is the contract. + +--- + +## 3. vLLM: usable, with sharp edges + +vLLM ships an official Gemma 4 recipe and a `--tool-call-parser gemma4` +flag. One open caveat: + +- **[#39392](https://github.com/vllm-project/vllm/issues/39392)**: + `` token floods under concurrent requests (2/5 fail at concurrency + 5; 0/5 sequential). Reporter suspects shared mutable parser state — + not thread-safe. Workaround is a global serialization lock around the + tool-call path. + +vLLM also still suffers the heterogeneous-attention-head Triton fallback +(LOW-severity entry in GOTCHAS.md) — throughput is well below what +similarly-sized models get on the same hardware. + +--- + +## 4. The `think` flag, properly characterized + +The April corpus had two contradictory entries: "always `think: false` +for single-turn JSON" and "never `think: false` for multi-turn tool loops +because it kills 26B." Both pointed at the same symptom (empty content, +no usable response) with different prescriptions. Field evidence from two +independent agents running `gemma4:26b` makes the actual interaction +clearer. + +### The two failure paths + +**Path A — `think: True` + `num_predict` too small.** +The hidden reasoning trace consumes the entire generation budget, leaving +`message.content=""` and `done_reason="length"`. Observed against +`num_predict=1024` with a complex system prompt + tools schema; thinking +trace alone was reaching the cap. + +**Path B — `think: False` + (long tool args OR multi-turn agentic loop +OR system prompt present).** Model emits a tool call on the wire that the +parser fails to extract, or emits near-immediate EOS at a decision turn. +Surface symptom: `tool_calls=[]`, empty content, sometimes a stray +`` token. Per Ollama #15539 this is parser-side; per the April +bakeoff it also shows up in deep multi-turn loops where each decision +turn is short. + +### The recipe + +| Loop shape | `think` | `num_predict` | Why | +|------------|---------|---------------|-----| +| Single-turn structured JSON | `False` | ≥ 2048 | No tool-call parsing involved; thinking adds latency without benefit; small budgets are safe | +| Short tool args, ≤ 5-step loop | `False` | ≥ 1024 | Field-verified to fire reliably at temp ≤ 0.5 with explicit Trigger→Action prompt structure | +| Long tool args (image-gen prompts, file payloads, code blocks) | `True` | ≥ 2048 | Field-verified: `think: False` produces empty `tool_calls` on the same context where `think: True` produces a proper call | +| Deep multi-turn agentic (≥ 10 steps) | `True` | ≥ 2048 | Per April bakeoff: `think: False` produces silent EOS at decision turns | + +**Safety floor:** if you can afford the latency, `think: True` + +`num_predict: 2048` works for *all* shapes above. The reason to deviate +is one of: +- Latency budget < ~3s per turn (thinking adds 1–2s) +- Predictability is critical (no hidden token generation) +- Schema-constrained output that excludes thinking tokens + +**Build a probe.** Both agents that diagnosed this in production wrote a +probe script that called the same Ollama endpoint with each flag setting +against a known-good message state and asserted `tool_calls != []`. If +you're flipping `think`, write the probe first — same-context comparisons +are how the parser-side failure surfaces. + +### Reproducing the parser-side diagnosis + +To verify whether your harness is hitting Ollama #15539 vs a different +failure path: + +```python +# Same model state, four conditions +combos = [ + {"system": None, "think": True}, + {"system": None, "think": False}, + {"system": "...", "think": True}, + {"system": "...", "think": False}, +] +for c in combos: + response = ollama.chat(model="gemma4:26b", messages=msgs(c), tools=TOOLS, + think=c["think"]) + print(c, "→ tool_calls:", len(response["message"]["tool_calls"])) +``` + +If only `system + think:False` produces zero tool calls, you've reproduced +the parser bug, not the model defect. + +--- + +## 5. The "doesn't fire tools without explicit ask" pattern + +This is the dominant production failure mode for small/mid agentic models +as of May 2026 — the model is capable of using tools, calls them when +explicitly told ("look this up", "search for"), and defaults to +answering-from-training otherwise. Now characterized in the literature: + +### Quantified + +- **["To Call or Not to Call"](https://arxiv.org/html/2605.00737v1)** + measures under/over-calling across model families. **Gemma3-27B over-calls + (462 vs 297 optimal)**; Mistral3.1-24B severely under-calls. Gemma 4 not + directly benched but the family signal is toward over-calling under + reasoning-mode conditions. The "doesn't fire" observation in production + likely reflects `think: false` runs where the model goes straight to + answer without the reasoning step that would surface the tool-call + decision. +- **[BiasBusters](https://arxiv.org/pdf/2510.00307)** quantifies + tool-selection bias driven by description metadata and position. Tools + with **edited** descriptions get **10× more usage** than identical tools + with original descriptions (GPT-4.1, Qwen2.5-7B). The wording of your + tool descriptions is itself tuning firing rate; this is not a marginal + effect. + +### What actually moves the needle + +**Probe & Prefill** ([arxiv 2605.09252](https://arxiv.org/html/2605.09252v1)) +— the most concrete fix in the recent literature, and the one most likely +to apply to a Gemma-4-sized agent. + +Core finding: **tool-call decisions are linearly decodable from the +model's pre-generation hidden states with AUROC 0.89–0.96** — +substantially outperforming the model's own verbalized reasoning about +whether to call a tool. The technique: + +1. Collect ~900 examples labeled "should call tool" vs "should not." +2. Train a logistic regression on the final-layer hidden state at the + pre-generation position. Trains in seconds on CPU. < 1 ms inference + overhead. +3. At inference, if the probe says "should call," prefill a short + steering sentence ("I'll use a tool for this. ") to bias decoding + toward the tool path. + +Demonstrated **48% reduction in unnecessary tool calls with 1.7% accuracy +loss** on Qwen3 (1.7/4/14/32B) and Llama (3.1-8B, 3.3-70B). Not tested +on Gemma 4 — but the technique is model-agnostic and the hidden-state +signal has held across every architecture they tried. + +**This is the only technique in the May 2026 literature that beats +careful prompt engineering on this specific problem.** Worth a serious +experiment for any non-trivial Gemma 4 agent. + +### Prompt-engineering techniques that help (but won't fully solve) + +- **Imperative tool descriptions with activation triggers** beat passive + descriptions. "Call this when the user asks about current events" + fires more reliably than "Search the web for current information." + (BiasBusters' 10× finding is the upper bound; expect smaller effects + in practice.) +- **Trigger → Action contracts in the system prompt** ("If the user + attempts a translation → call `score_vocab`") outperform softer + guidance. Field-verified to take a Gemma 4 26B agent from ~0% + tool-firing at default temperature to ~80% at temp 0.4. +- **Lower temperature** (0.3–0.5) increases firing rate on agentic + loops. Default Ollama temperature (~0.8) biases toward + conversational mimicry of recent chat history. +- **Reactive-mode nudge loop**: if the model emits text without calling + a tool on a turn where tools are available, append a system message + ("Was a tool needed? If yes, call it.") and re-prompt. One agent in + the field uses this in its autonomous-research loop but not its + reactive chat loop; the reactive loop has the firing-rate problem + the autonomous loop doesn't. + +### Benchmarking your fix + +[NVIDIA's When2Call](https://github.com/NVIDIA/When2Call) (NAACL 2025) is +the relevant eval: 1000 MCQs scoring "answer directly / call tool / ask +clarifying / decline." Their finding: **RPO consistently > SFT** for +training models to make this decision well. Useful as the harness even +if you're not training, because it gives you a single number to track +across prompt / parameter changes. + +--- + +## 6. Two-stage tool routing — still alive and well + +In-model selection has **not** caught up to two-stage routing for the +"should-call" decision as of May 2026. The router pattern has shifted +from "another LLM" to "small classifier": + +- **[ToolCallVerifier](https://huggingface.co/llm-semantic-router/toolcall-verifier)**: + ModernBERT-based 0.1B token classifier. Recall 0.92, precision 0.95, + F1 0.935. Designed for prompt-injection defense but the same pattern + works as a "should-call" gate. +- **vLLM Semantic Router (v0.1 Iris, Jan 2026)**: production framework + for taxonomy-guided routing + confidence-cascade. +- **Probe & Prefill** (above) is the limit case: the "router" is a + logistic regression on the *main model's own* hidden states. No + separate model to host. + +**Practical recommendation:** if you're under 5 tools and can't afford +to train a probe, Trigger → Action prompting + temp 0.4 + reactive nudge +loop will get you to ~70–80% firing rate. Beyond that, the probe technique +is the lowest-cost concrete improvement. + +--- + +## 7. Structured output: XGrammar-2 changes the math + +**[XGrammar-2](https://blog.mlc.ai/2026/05/04/xgrammar-2-fast-customizable-structured-generation)** +(MLC blog 2026-05-04, [arxiv 2601.04426](https://arxiv.org/abs/2601.04426)) +is the headline structured-output development of the year: + +- New "Structural Tag" composable JSON protocol uniformly expressing + OpenAI Harmony, tool calling, reasoning channels, and custom output + structures +- Cross-grammar caching with ~50% structure reuse on 50-tool schemas +- Repetition state compression: 534ms → 5.37ms (100×) +- Batching + speculative-decoding support +- **Default structured-generation backend for vLLM, SGLang, TensorRT-LLM, + MLC-LLM** as of March 2026 + +The per-token overhead is now low enough for production. Smaller models +showed "substantial gains in output accuracy" on BFCL-V3. + +**Important distinction:** structured output forces the **format** once +you've decided to call a tool. It does **not** solve "decide whether to +call a tool." For that, see § 5. + +Ollama's native `format` schema field (not the broken `format: "json"`) +remains the right Ollama-native option; community reports it's more +reliable than free-form JSON prompting but still leaks for some model +families. No Gemma-4-specific reliability data yet. + +--- + +## 8. New Gemma 4 variants — slim pickings + +No community fine-tune released to date materially improves tool-calling +reliability over base Gemma 4 26B-it. Notable releases since April: + +| Release | What | Tool-calling relevance | +|---------|------|------------------------| +| [`google/gemma-4-26B-A4B-it-assistant`](https://huggingface.co/google/gemma-4-26B-A4B-it-assistant) | 0.4B drafter for speculative decoding | Generation speed only — not a tool-calling tune | +| `nvidia/Gemma-4-26B-A4B-NVFP4` | NVFP4 quant for Blackwell | Hardware target, not a tune | +| [`Jackrong/Gemopus-4-26B-A4B-it-GGUF`](https://huggingface.co/Jackrong/Gemopus-4-26B-A4B-it-GGUF) | SFT for style consistency / Markdown structure | **Explicitly disclaims tool calling**: "known compatibility issues, not unique to this model" | +| HF blog / TRL fine-tuning recipe | Fine-tune `gemma-4-E2B-it` for tool calling on H100 | Recipe, not a released model | +| Unsloth GGUFs | All sizes, plus fine-tuning guide | No agentic tune | + +If you need a robust local agentic small model **today**, the field +still recommends **Qwen3-Coder** family. Gemma 4 is workable for agentic +use with the harness work above, but it's not the path of least +resistance for that workload. + +--- + +## 9. What this means for an existing harness + +A concrete checklist if you have a Gemma 4 26B agent in production and +are debugging tool-call reliability: + +1. **Check your Ollama version.** Below 0.22.1, upgrade. Below 0.20.6, + you're missing the first round of Gemma 4 tool-call fixes entirely. +2. **Write the four-condition probe** from § 4 against a known-good + message state. If only `system + think:False` zeros out + `tool_calls`, you have #15539, not a model defect. +3. **Set the recipe** from § 4 by loop shape. The safe combo for almost + all cases is `think: True` + `num_predict: 2048`. +4. **Audit your tool descriptions** for activation verbs. "Call this + when…" beats "Search for…" by a non-marginal margin per BiasBusters. +5. **Lower temperature to 0.4** if you're at default. This alone took + one production agent from ~0% to ~80% firing on triggers. +6. **Add a reactive-mode nudge loop**: if `tool_calls=[]` on a turn + where the user asked something factual, re-prompt with a system + message asking whether a tool was needed. +7. **If still under-firing**, prototype Probe & Prefill on ~900 labeled + turns from your own log. This is the lowest-effort high-impact + technique in the current literature. +8. **Consider Qwen3-Coder for the tool-heavy path** if you're not + committed to Gemma 4 for other reasons (vision, multilingual, + licensing). + +## Contradictions with prior corpus, flagged + +- `GOTCHAS.md` HIGH: "`think: false` Kills Gemma 4 26B in Multi-Turn + Tool-Calling Loops" — **partially superseded**. The failure is real + but is parser-side in Ollama (#15539), not model-side. Same fix + (use `think: True`) but the root cause framing was wrong, and the + failure also affects e4b, not just 26B. +- `SYNTHESIS.md` "Mandatory Ollama Settings": the single-turn + "always `think: false`" rule is still correct *for single-turn JSON + pipelines*. The multi-turn agent guidance needs the recipe from § 4 + applied. +- `GOTCHAS.md` MEDIUM: "Tool Calling Broken in Ollama v0.20.0 Streaming" + — partially fixed (v0.20.6 + v0.22.1). Re-test on current version + before assuming streaming tool calls are still broken. +- `GOTCHAS.md` LOW: "`` Token Infinite Loop (Vulkan backends)" + — fixed in llama.cpp b8691+. + +## Sources + +Primary: +- [ollama/ollama#15539](https://github.com/ollama/ollama/issues/15539) +- [ollama/ollama releases](https://github.com/ollama/ollama/releases) +- [ggml-org/llama.cpp#21316](https://github.com/ggml-org/llama.cpp/issues/21316), + [#21321](https://github.com/ggml-org/llama.cpp/issues/21321), + [#22080](https://github.com/ggml-org/llama.cpp/issues/22080) +- [vllm-project/vllm#39392](https://github.com/vllm-project/vllm/issues/39392) + +Research: +- ["LLM Agents Already Know When to Call Tools" — Probe & Prefill](https://arxiv.org/html/2605.09252v1) +- ["To Call or Not to Call"](https://arxiv.org/html/2605.00737v1) +- [BiasBusters](https://arxiv.org/pdf/2510.00307) +- [NVIDIA When2Call](https://github.com/NVIDIA/When2Call) +- [XGrammar-2 paper](https://arxiv.org/abs/2601.04426), + [MLC blog](https://blog.mlc.ai/2026/05/04/xgrammar-2-fast-customizable-structured-generation) + +Community: +- [Tim Gregg: llama.cpp Gemma 4 thinking-prompt fix](https://complete.tech/blog/llamacpp-gemma4-thinking-prompt-local-fix/) +- [unsloth/gemma-4-26B-A4B-it-GGUF discussion #2 (unused49 root cause)](https://huggingface.co/unsloth/gemma-4-26B-A4B-it-GGUF/discussions/2) +- [google/gemma-4-26B-A4B-it discussion #10 (thinking-mode behavior)](https://huggingface.co/google/gemma-4-26B-A4B-it/discussions/10) diff --git a/GOTCHAS.md b/GOTCHAS.md index 8d33eff..68e61de 100644 --- a/GOTCHAS.md +++ b/GOTCHAS.md @@ -2,6 +2,13 @@ > Derived from Seth's production implementations (Simon, AI_Visualizer) > and community reports. These are hard-won lessons. +> +> **2026-05-25 corrections:** several entries below were corrected by +> field evidence from two independent agents and by Ollama/llama.cpp +> issue-tracker review. See `CORPUS_tool_calling_2026-05.md` for the +> consolidated current state. The "`think: false` kills 26B" rule and +> the `` Vulkan-loop entry both need re-reading against that +> update — they are marked **SUPERSEDED** in place below. ## CRITICAL: Thinking Mode Eats Context (single-turn pipelines only) @@ -75,10 +82,32 @@ Ollama defaults `num_predict` to 128 tokens. Almost any useful Gemma 4 output ex **Fix:** Always set `num_predict` explicitly. Minimum recommended: 512. For JSON output: 2048+. -## HIGH: `think: false` Kills Gemma 4 26B in Multi-Turn Tool-Calling Loops +## HIGH: `think: false` Kills Gemma 4 26B in Multi-Turn Tool-Calling Loops [SUPERSEDED 2026-05-25] **Severity: HIGH — silent agent-loop failure. Setting is what the old guidance said to do.** +> **SUPERSEDED 2026-05-25:** the symptom is real but the diagnosis was +> wrong. Per [ollama/ollama#15539](https://github.com/ollama/ollama/issues/15539), +> the failure is an **Ollama parser bug** triggered by the combination +> `system prompt + think:false + tools` — the model emits a valid tool +> call on the wire, the framework returns `tool_calls=[]` with an empty +> `content` and a stray `` token. **The bug affects `gemma4:e4b` +> too, not just 26B MoE.** Field evidence from a production agent at +> `num_predict=1024` confirms `think: false` works fine for short tool +> arguments and shallow loops — the original bakeoff hit the deep-loop +> case where the parser fails. +> +> **Replacement rule:** see `CORPUS_tool_calling_2026-05.md` § 4 for the +> proper recipe by loop shape. Safe default for all cases: +> `think: True` + `num_predict ≥ 2048`. Diagnostic for whether you're +> hitting this bug: run a four-condition probe (system × think) at the +> same model state — if only `system + think:False` zeros tool_calls, +> you've reproduced #15539. +> +> The April-2026 content below is preserved for the historical record but +> should not be applied without first reading the May update. + + Reproduced on 2026-04-18 against `gemma4:26b` via Ollama 0.20.4 on a 3090 Ti (steel141). Contradicts the older "always think:false" guidance (see § "Thinking Mode Eats Context" below — now scoped to single-turn pipelines only). @@ -178,9 +207,9 @@ Community-reported: Flash Attention causes Gemma 4 31B Dense to hang indefinitel **Fix:** Use 26B for long prompts, or disable Flash Attention if running 31B on affected hardware. -## MEDIUM: Tool Calling Broken in Ollama v0.20.0 Streaming +## MEDIUM: Tool Calling Broken in Ollama v0.20.0 Streaming [PARTIALLY FIXED] -**Severity: MEDIUM — version-specific** +**Severity: MEDIUM — version-specific. PARTIALLY FIXED in v0.20.6 + v0.22.1.** As of early April 2026, Gemma 4 tool calling has issues in Ollama v0.20.0: the tool call parser fails and streaming drops tool calls entirely. Community reports include format mismatches and continuous loops in llama.cpp / LM Studio. @@ -188,6 +217,17 @@ As of early April 2026, Gemma 4 tool calling has issues in Ollama v0.20.0: the t **Fix:** Use non-streaming for tool calls (Simon does this). Test tool calling thoroughly when upgrading Ollama versions. Seth's implementations work reliably with non-streaming tool calls. +> **2026-05-25 update:** Ollama [v0.20.6](https://github.com/ollama/ollama/releases/tag/v0.20.6) +> explicitly improved Gemma 4 tool-calling and added "improved parallel +> tool calling for streaming responses." [v0.22.1](https://github.com/ollama/ollama/releases/tag/v0.22.1) +> updated the Gemma 4 renderer for thinking + tool-calling improvements. +> Current stable is **v0.24.0** (2026-05-14). Re-test streaming on current +> versions before assuming the original bug persists. **Open bugs** +> remain — see `CORPUS_tool_calling_2026-05.md` § 1 for the current list, +> especially [#15539](https://github.com/ollama/ollama/issues/15539) +> (`system + think:false + tools` returns empty tool_calls) which is the +> most likely failure mode you'll hit on a recent Ollama. + ## MEDIUM: VRAM-Hungry for Context **Severity: MEDIUM — affects hardware planning** @@ -222,14 +262,24 @@ Heterogeneous attention head dimensions in Gemma 4 force vLLM to fall back to a **Fix:** Use Ollama instead of vLLM for now, or wait for the fix. -## LOW: `` Token Infinite Loop (Vulkan backends) +## LOW: `` Token Infinite Loop (Vulkan backends) [FIXED] -**Severity: LOW — Vulkan-specific** +**Severity: LOW — Vulkan-specific. FIXED in llama.cpp b8691+.** Gemma 4 can generate `` or `` tokens in an infinite loop on Vulkan backends in llama.cpp. **Source:** [ggml-org/llama.cpp#21516](https://github.com/ggml-org/llama.cpp/issues/21516) +> **2026-05-25:** Per the +> [unsloth/gemma-4-26B-A4B-it-GGUF discussion thread](https://huggingface.co/unsloth/gemma-4-26B-A4B-it-GGUF/discussions/2), +> the `` / channel-token flood across CUDA / ROCm / Vulkan / SYCL +> was a llama.cpp eval bug, **not** a quantization problem. Fixed in +> b8691+. Old quants work fine on patched builds. +> +> Caveat: 31B + f16/f32 vision projector still produces `` +> floods on current builds per one report — the eval-bug fix didn't +> cover the multimodal projector path. + ## MEDIUM: `google/gemma_pytorch` Abandoned for Gemma 4 **Severity: MEDIUM — wastes time on a dead-end path** diff --git a/README.md b/README.md index 169804b..69956ed 100644 --- a/README.md +++ b/README.md @@ -14,6 +14,8 @@ Research corpus and implementation guidance for Google Gemma 4, based on product | `CORPUS_capabilities.md` | Modalities (vision, audio, video, tools), what it can/can't do | When scoping what Gemma 4 can handle | | `CORPUS_benchmarks.md` | Full benchmark table vs Gemma 3, arena scores, agentic scores | When comparing Gemma 4 to alternatives | | `CORPUS_tool_calling_format.md` | Native token format + JSON API format for function calling | When implementing tool calling | +| `CORPUS_tool_calling_2026-05.md` | **May 2026 update — read first for tool-call debugging.** Ollama parser fixes (#15539), `think` flag properly characterized, the "doesn't fire tools without explicit ask" research (Probe & Prefill, BiasBusters, When2Call), XGrammar-2. Supersedes parts of `GOTCHAS.md` (think:false rule, Vulkan loop, streaming bug all updated in-place with pointers here) | When debugging tool-call reliability, deciding `think` value, or planning a router/probe layer | +| `CORPUS_agentic_retrieval.md` | Post-RAG landscape — hybrid retrieval + cross-encoder rerank as the settled base, CRAG-shaped flows in LangGraph/LlamaIndex, GraphRAG (LightRAG/LazyGraphRAG), memory architectures (mem0 vs Letta MemFS), deep-research agents (gpt-researcher, smolagents), production-vs-experimental sorting | When stepping up from basic embedding RAG to agentic retrieval, picking a memory layer, or scoping a deep-research agent | | `CORPUS_cli_coding_agent.md` | Positioning Gemma 4 for CLI coding agent use (openclaw / open code / pi / hermes / aider style). Honest take on what Google did and didn't measure, head-to-head with `qwen3-coder:30b`, homelab setup pointer | When scoping a CLI coding agent or deciding Gemma 4 vs Qwen3-Coder | | `docs/openwebui-setup.md` | How to configure Gemma 4 inside OpenWebUI — per-setting reference, two ready-to-bake Workspace Model profiles (chat + extract), and a symptom→cause troubleshooting table mapped back to GOTCHAS.md. Assumes Ollama + OpenWebUI are already running. | When setting up or debugging a Gemma 4 model in OpenWebUI, or handing the front-end config to someone else | | `docs/reference/bakeoff-2026-04-18.md` | CLI-coding-agent bakeoff on 3090 Ti. **Rounds 1/2 misidentified the cause; Round 3 (the correct one): `think: false` silent-stops gemma4:26b at certain multi-turn states on 32K context.** 31B and Qwen3-Coder robust to the flag. Harness at `scripts/bakeoff/` | When deciding which model to back a CLI agent with, writing a custom agent payload, or debugging a silent tool-call halt |