Seth/gemma4-research
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

docs: session handoff — Gemma 4 tool-calling May-2026 update + bot audit

Browse Source 
Captures the session that produced 438a96f (two new corpus docs + GOTCHAS
corrections) plus the gitignored private/ folder with mort-bot and
hola-bot recommendations. Documents the deliberate-contradiction-in-GOTCHAS
approach, the Ollama #15539 finding, and the cross-bot think-flag
resolution.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Mortdecai
2026-05-25 10:34:21 -04:00
parent 438a96f235
commit f11aa31c8a
1 changed files with 189 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.claude/handoffs/2026-05-25-103131-gemma4-tool-calling-may-update.md
+189
View File
@@ -0,0 +1,189 @@
# Handoff: Gemma 4 Tool-Calling May-2026 Update + Bot Audit
## Session Metadata
- Created: 2026-05-25 10:31:31
- Project: /home/claude/bin/gemma4-research
- Branch: master
- Session duration: ~2 hours
### Recent Commits (for context)
- 438a96f docs: May 2026 tool-calling update + agentic retrieval landscape
- 0f82cd7 docs: session handoff — GPU bakeoff (3090 Ti vs Strix Halo)
- 91842f3 docs: scrub PII/IPs from gpu-bakeoff
- 22af597 docs: remove V100 from GPU bakeoff
- b619035 feat: GPU bakeoff — 3090 Ti vs V100 vs Strix Halo
## Handoff Chain
- **Continues from**: [2026-04-20-055658-gpu-bakeoff-3090-vs-strix.md](./2026-04-20-055658-gpu-bakeoff-3090-vs-strix.md)
- Previous title: GPU Bakeoff — 3090 Ti vs Strix Halo (+ parked native-bakeoff scaffold)
- **Supersedes**: None. This is a fresh thread (tool-calling research) on top of the existing corpus, not a continuation of the GPU bakeoff work.
> The chain link above is structural only — the GPU bakeoff and this update don't share content.
## Current State Summary
Seth asked for two things: (1) a deep look at why `mort-bot` and `hola-bot` struggle to fire tools when not explicitly asked, and (2) web research on what's new in agentic retrieval beyond RAG. Both bots run `gemma4:26b` against Ollama. The result is two new public corpus docs (committed and pushed to git.sethpc.xyz/Seth/gemma4-research), three corrections marked in-place in `GOTCHAS.md`, and a `private/` folder (gitignored) with bot-specific recommendations. The biggest finding is that the existing corpus's "`think: false` kills 26B in multi-turn tool loops" rule is half-wrong — the failure is real but it's the **Ollama parser** bug ([#15539](https://github.com/ollama/ollama/issues/15539)) triggered by `system + think:false + tools`, not a model-side defect. The two bots had A/B-tested `think` and landed on opposite values for genuinely different reasons; the new docs reconcile this.
## Codebase Understanding
### Architecture Overview
`gemma4-research/` is a research corpus, not an application. Tracked in Gitea
(`git.sethpc.xyz/Seth/gemma4-research`), follows the homelab convention of
push-on-commit. Three tiers of files:
- **Auto-load tier** (`README.md`, `SYNTHESIS.md`, `GOTCHAS.md`, `MEMORY.md`):
read every session, kept lean.
- **On-demand tier** (`CORPUS_*.md`, `docs/reference/*.md`): pulled in when a
task needs the detail. The `README.md` table has "when to read" hints.
- **Private** (`private/`, new): gitignored bot-specific findings that
reference mort-bot/hola-bot directly.
The corpus was last meaningfully updated 2026-04-12 (the "Seth Freiberg, 2026-04-12" timestamp in `SYNTHESIS.md`). The two new docs are the May-2026 supplement on top of that baseline.
### Critical Files
| File | Purpose | Relevance |
|------|---------|-----------|
| `CORPUS_tool_calling_2026-05.md` | **New, this session.** May-2026 state of Gemma 4 tool calling. Supersedes parts of the April corpus. | Read FIRST for any future tool-calling debugging. The recipe-by-loop-shape table in § 4 is the load-bearing artifact. |
| `CORPUS_agentic_retrieval.md` | **New, this session.** Post-RAG landscape, production-vs-experimental sorting | Read when designing retrieval / memory / deep-research for a new Gemma 4 agent. |
| `GOTCHAS.md` | Existing — three entries updated in-place this session | Three entries now have SUPERSEDED/FIXED/PARTIALLY FIXED markers with pointers to `CORPUS_tool_calling_2026-05.md`. Don't re-edit those without reading the May doc first. |
| `private/THINK_FLAG_RESOLUTION.md` | **New, gitignored.** Timeline of mort-bot and hola-bot's `think` flag A/B tests | Read before changing either bot's `think` setting. Has primary-source quotes from each bot's handoffs. |
| `private/MORT_BOT_RECOMMENDATIONS.md` | **New, gitignored.** 8 prioritized code-level suggestions with file:line refs | Take to `~/bin/mort-bot/` next session. Ordered quick-wins → experiments. |
| `private/HOLA_BOT_RECOMMENDATIONS.md` | **New, gitignored.** 7 prioritized suggestions for `~/bin/hola-bot/` | Take to `~/bin/hola-bot/` next session. Includes cross-bot coordination notes for the shared steel141 Ollama. |
| `README.md` | Two new index rows added for the two new corpus docs | Don't add more without re-reading — the table has carefully chosen "when to read" clauses. |
### Key Patterns Discovered
- **Scrubbed-public + named-private split** works cleanly here. The public docs talk about "a production agent" / "field observations" without naming bots, IPs, or services. The private folder uses real names. The `private/README.md` documents the split convention.
- **In-place GOTCHAS corrections with SUPERSEDED markers** preserve historical record while pointing forward. This is friendlier than deleting + rewriting because future readers (including the maintainer rereading their own notes) can see what was thought to be true vs what's true now.
- **Push-on-commit convention** for `~/bin/*` Gitea projects (memory: `feedback_gitea_push_on_commit.md`) — `git commit` and `git push` bundled in one tool call. Followed this session.
## Work Completed
### Tasks Finished
- [x] Audited `~/bin/mort-bot/` tool-calling architecture (via Explore subagent — read llm.py, bot.py, discord_bot.py, personality.txt, config.py, CLAUDE.md, DECISIONS.md, 5 recent handoffs)
- [x] Audited `~/bin/hola-bot/` tool-calling architecture (via Explore subagent — read tools.py, llm.py, bot.py, personality.txt, config.py, recent handoffs)
- [x] Web research on Gemma 4 tool-calling state-of-the-art (general-purpose subagent, ~325s, 47 tool uses)
- [x] Web research on post-RAG agentic retrieval (general-purpose subagent, ~227s, 24 tool uses)
- [x] Reconstructed the `think` flag timeline for both bots from git log + handoffs
- [x] Wrote `CORPUS_tool_calling_2026-05.md` (10 sections, scrubbed)
- [x] Wrote `CORPUS_agentic_retrieval.md` (10 sections, scrubbed)
- [x] Updated `GOTCHAS.md` with three SUPERSEDED/FIXED markers + pointers
- [x] Updated `README.md` with two new index rows
- [x] Added `private/` to `.gitignore`
- [x] Wrote private bot-specific notes (README, THINK_FLAG_RESOLUTION, MORT_BOT_RECOMMENDATIONS, HOLA_BOT_RECOMMENDATIONS)
- [x] Verified `private/` is gitignored before staging
- [x] Scanned new public files for PII leaks (none introduced)
- [x] Committed + pushed in a single step per the homelab convention
### Files Modified
| File | Changes | Rationale |
|------|---------|-----------|
| `.gitignore` | Added `private/` entry | Bot-specific notes shouldn't be public |
| `CORPUS_tool_calling_2026-05.md` | New file, 10 sections, ~750 lines | The May supplement to the April corpus |
| `CORPUS_agentic_retrieval.md` | New file, 10 sections, ~230 lines | First post-RAG content in the corpus |
| `GOTCHAS.md` | Three entries marked SUPERSEDED/FIXED/PARTIALLY FIXED with pointers | The April-corpus contradictions surfaced this session needed in-place flags |
| `README.md` | Two new index rows | New docs need indexing |
| `private/README.md` | New (gitignored) | Index of the private folder |
| `private/THINK_FLAG_RESOLUTION.md` | New (gitignored) | The two-bot `think` flag timeline + recommendations |
| `private/MORT_BOT_RECOMMENDATIONS.md` | New (gitignored) | 8 prioritized recommendations for mort-bot with file:line refs |
| `private/HOLA_BOT_RECOMMENDATIONS.md` | New (gitignored) | 7 prioritized recommendations for hola-bot |
### Decisions Made
| Decision | Options Considered | Rationale |
|----------|-------------------|-----------|
| Two separate corpus docs (tool-calling + retrieval) instead of one combined "May 2026 update" | (a) combined doc, (b) two separate | Different read-when triggers, different audiences. Tool-calling updates existing corpus; retrieval is brand-new topic. Two docs scale better. |
| Mark GOTCHAS entries SUPERSEDED in-place with pointer to new file, instead of deleting/rewriting | (a) delete + rewrite, (b) SUPERSEDED in place, (c) leave alone and rely on the new file | In-place preserves historical record + warns future readers. Deletion would let stale advice survive in someone's grep cache or printed reference. |
| Scrubbed public + named private split, with `private/` gitignored | (a) public-only with all PII scrubbed, (b) two folders | The bot-specific recommendations need real names + file paths to be actionable. Splitting cleanly maps to Seth's instruction. |
| Push-on-commit (no review checkpoint) for the public docs | (a) commit + ask before push, (b) commit + push together | Followed `~/bin/*` Gitea convention (memory `feedback_gitea_push_on_commit.md`). Public docs are not destructive; the convention exists precisely for this case. |
| Do NOT touch `SYNTHESIS.md`'s "Mandatory Ollama Settings" section even though it has the same single-turn/multi-turn split logic that needs the new recipe applied | (a) update SYNTHESIS.md too, (b) leave alone, point from CORPUS_tool_calling_2026-05.md | SYNTHESIS.md is Seth's voice ("Seth Freiberg, 2026-04-12") — editing it without his sign-off felt wrong. The new corpus doc explicitly calls out the SYNTHESIS section needing the recipe applied. Leave for him to update. |
## Pending Work
## Immediate Next Steps
1. **Take the private recommendations to the bot projects.** `cd ~/bin/mort-bot && cat ~/bin/gemma4-research/private/MORT_BOT_RECOMMENDATIONS.md` — then triage which of the 8 items to do this week. The top-3 quick wins (tool-description rewrite, reactive-loop nudge, temperature drop to 0.4) are each half-day or less and have the biggest expected effect.
2. **Same for hola-bot:** `cat ~/bin/gemma4-research/private/HOLA_BOT_RECOMMENDATIONS.md`. The top-1 here is fixing `remember_fact` (the confirmed gap from the 2026-05-25 handoff — `user_memory` table is empty in production).
3. **Optional: have Seth review `SYNTHESIS.md` § "Mandatory Ollama Settings"** against the new recipe in `CORPUS_tool_calling_2026-05.md` § 4. The synthesis section still has the old single-turn/multi-turn split that needs the parser-side framing applied. I did not touch it because it's signed.
### Blockers/Open Questions
- [ ] **Is mort-bot's steel141 Ollama actually still on 0.20.4?** The corpus baseline says so, but no one verified it this session. Could be older or newer. `ssh steel141 'ollama --version'` to check. Several Gemma 4 tool-call fixes shipped between 0.20.4 and current stable (0.24.0); upgrading may itself improve firing rate.
- [ ] **Does hola-bot use a proxy in front of Ollama?** If yes, watch [ollama/ollama#15719](https://github.com/ollama/ollama/issues/15719) (LiteLLM proxy + Gemma 4 infinite tool-call loop on 0.20.6/0.20.7). Likely not — the audit said direct Ollama calls — but worth confirming before any upgrade.
### Deferred Items
- **Probe & Prefill experiment** (arxiv 2605.09252): the most concrete tool-firing fix in the May literature. Linear probe on Gemma's pre-generation hidden states predicts "should call tool" at AUROC 0.89–0.96. Catch: Ollama doesn't expose hidden states, so this requires vLLM or transformers routing. Deferred to a hypothetical Q3 2026 experiment.
- **Hola-bot: bump `num_predict` 1024→2048 + flip `think: True`** to put it on the same safe combo as mort-bot. Deferred until hola-bot expands its tool set with long-arg tools — current `think: False` shape works.
- **`SYNTHESIS.md` update** — see Immediate Next Steps #3.
- **Add observability for tool-firing rate to both bots.** Both already log `tool_calls=N` per turn; the deferred work is aggregating into a daily metric. Hola-bot's `MORT_BOT_RECOMMENDATIONS.md` and `HOLA_BOT_RECOMMENDATIONS.md` both list this as a low-priority item.
## Context for Resuming Agent
## Important Context
**The corpus has a deliberate contradiction now — treat it as a feature, not a bug.** `GOTCHAS.md` still has the April-2026 rule "`think: false` kills Gemma 4 26B in multi-turn tool-calling loops" *with* a SUPERSEDED block on top pointing to the May correction. If you're debugging tool-calling and reach for GOTCHAS, you'll see both views in one place. Don't reconcile by deleting the old content — Seth explicitly tracks superseded knowledge to prevent re-debating. Same pattern for the Vulkan `<unused>` loop entry (now FIXED in llama.cpp b8691+) and the Ollama 0.20.0 streaming bug (PARTIALLY FIXED in v0.20.6 + v0.22.1).
**The two bots resolved the `think` flag in opposite directions for genuinely different reasons.** Don't assume one is wrong. See `private/THINK_FLAG_RESOLUTION.md` for the timeline reconstruction with primary-source quotes from each bot's git history and handoffs. The unified rule that fits both observations is in `CORPUS_tool_calling_2026-05.md` § 4: safe combo is `think: True` + `num_predict ≥ 2048`. Deviations are legitimate but should be empirically tested with a four-condition probe.
**Ollama [#15539](https://github.com/ollama/ollama/issues/15539) is the central technical finding of the session.** The bug triggers on `system prompt + think:false + tools` — model emits a valid tool call on the wire, parser returns `tool_calls=[]`. Affects `gemma4:e4b` too, not just 26B MoE. This explains both bots' empirical findings cleanly. If you see a Gemma 4 agent silently dropping tool calls, the diagnostic is the four-condition probe in § 4 of the new doc.
**The "doesn't fire tools without explicit ask" failure pattern is now research-characterized.** Three relevant papers: Probe & Prefill (arxiv 2605.09252) — hidden-state probe beats prompt engineering, AUROC 0.89–0.96; BiasBusters (arxiv 2510.00307) — tool descriptions with activation triggers fire 10× more than passive descriptions; NVIDIA When2Call (NAACL 2025) — benchmark + RPO training method. These are the cited fixes in `MORT_BOT_RECOMMENDATIONS.md` and `CORPUS_tool_calling_2026-05.md` § 5.
**Public commit is on master and pushed.** Commit `438a96f` (`docs: May 2026 tool-calling update + agentic retrieval landscape`). 5 files, 797 insertions. The `private/` folder is on disk but gitignored — if you `git status` it should NOT appear in modified/untracked.
### Assumptions Made
- The corpus's "April 2026" baseline is approximately current — only ~6 weeks have passed since the 04-10 timestamps, not the "several months" Seth said in his prompt. Worked the actual timeline, not the felt timeline.
- The Explore-subagent's file:line citations from the bot audits are accurate. Spot-checked a few (e.g. `llm.py:549-550` for the autonomous nudge) against the bots directly. If you act on a specific recommendation, re-verify the file:line before editing — bots may have moved since the audit.
- Seth wants experimental things tried on the bots ("Both bots are experimental and in development, so if something experimental is available, these are the bots to try it with"). The recommendations are aggressive about suggesting changes; tone them down if you're being more conservative.
- The push-on-commit convention applies (per memory `feedback_gitea_push_on_commit.md`). Verified before pushing.
### Potential Gotchas
- **`private/` is on the same filesystem as the public corpus.** If you accidentally `git add .` (vs `git add <specific>`), the gitignore should catch it — but verify with `git status` before committing.
- **The hola-bot audit was done by a subagent and uses some file:line numbers that may have shifted.** Hola-bot has been moving fast (recent commits 5/23, 5/24, 5/25). Before acting on a specific line reference, `grep` the file to confirm.
- **Don't apply the new tool-calling recipe to the AI_Visualizer / single-turn JSON pipeline shape.** The original "always `think: false`" rule in `SYNTHESIS.md` is still correct *for single-turn JSON pipelines*. Only the multi-turn agent guidance was wrong. The new doc says this in § 4.
- **The `MEMORY.md` index file has TWO memories that may need updating** — `feedback_gitea_push_on_commit.md` (still accurate) and `feedback_scrub_pii_before_publish.md` (still accurate). The latter informed the public/private split this session.
- **mort-bot reads steel141 Ollama at 192.168.0.141:11434.** Hola-bot uses the same primary plus pve197 as fallback. If you upgrade Ollama on steel141, you affect both bots simultaneously — coordinate (see `HOLA_BOT_RECOMMENDATIONS.md` § "Cross-bot notes").
## Environment State
### Tools/Services Used
- **WebSearch / WebFetch** — used by the two general-purpose research subagents.
- **Explore subagent** — used for both bot audits (read-only; no edits).
- **gitea push** — public commit pushed to `git.sethpc.xyz/Seth/gemma4-research`.
- **TaskCreate / TaskUpdate** — 5 tasks tracked through the session.
### Active Processes
- None spawned by this session. mort-bot's Docker container (CT 652 on node-241), hola-bot's systemd service, the steel141 Ollama, and pve197 Ollama all running independently — none touched this session.
### Environment Variables
- None set / read this session.
## Related Resources
- **Commit pushed:** [git.sethpc.xyz/Seth/gemma4-research/commit/438a96f](https://git.sethpc.xyz/Seth/gemma4-research/commit/438a96f)
- **Primary new docs:** `CORPUS_tool_calling_2026-05.md`, `CORPUS_agentic_retrieval.md`
- **Private notes:** `private/THINK_FLAG_RESOLUTION.md`, `private/MORT_BOT_RECOMMENDATIONS.md`, `private/HOLA_BOT_RECOMMENDATIONS.md`
- **Key external sources:**
- [ollama/ollama#15539](https://github.com/ollama/ollama/issues/15539) — the parser bug central to the session
- [Probe & Prefill paper (arxiv 2605.09252)](https://arxiv.org/html/2605.09252v1)
- [BiasBusters (arxiv 2510.00307)](https://arxiv.org/pdf/2510.00307)
- [XGrammar-2 (MLC blog)](https://blog.mlc.ai/2026/05/04/xgrammar-2-fast-customizable-structured-generation)
- [Microsoft LazyGraphRAG](https://www.microsoft.com/en-us/research/blog/lazygraphrag-setting-a-new-standard-for-quality-and-cost/)
- **Bot source projects (for next session):**
- `~/bin/mort-bot/` — main file `llm.py`, key handoff `2026-05-20-225512-diag-3-fixes-and-keepalive-30m.md`
- `~/bin/hola-bot/` — main file `llm.py`, key handoff `2026-05-25-091134-whitelist-refactor-3-students-onboarded.md`
---
**Security Reminder**: Before finalizing, run `validate_handoff.py` to check for accidental secret exposure.