fix: walk back round-1/2 conclusions — the cause was think=false all along
Seth asked "was this with think=false?" Yes — and that was the only question that mattered. Everything I concluded in round 1 and round 2 was wrong. Actual cause, isolated in round 3: - At identical message state, gemma4:26b with think=false returns eval=4 (silent stop); with think unset or think=true, returns eval=165 and emits the correct tool call. - Original round-1 write_file harness + think unset: 26B passes in 8 iters, 20s. No mitigations needed. - 31B dense and qwen3-coder:30b tolerate think=false; 26B MoE does not. Red herrings (kept on-record in the bakeoff doc, not silently erased): - Round 1: "write_file tool-call argument size" — wrong - Round 2a: refuted the arg-size theory but for the wrong reason (still failed because think=false was still set) - Round 2b: "cumulative tool-response context size" — truncating did make 26B pass, but by coincidence. Shorter context at the decision turn dodged the think=false side effect. Why the existing "always think:false" guidance was misleading: it was derived from AI_Visualizer (single-turn JSON pipelines) where thinking tokens do eat num_predict invisibly. In multi-turn tool-calling agents the channels are separate and the flag has a different effect — catastrophic on 26B specifically. Doc updates: - GOTCHAS: replaced the 26B entry with the actual cause; scoped the original "Thinking Mode Eats Context" entry to single-turn pipelines - SYNTHESIS: split the "Mandatory Ollama Settings" block into single-turn vs multi-turn variants; updated anti-patterns and quick-start checklist - CORPUS_cli_coding_agent.md: revised pointer and config template - docs/reference/bakeoff-2026-04-18.md: added Round 3 section with the correction notice at the top of the file and full diagnostic methodology New artifacts: harness_no_think_flag.py, harness_write_no_think.py, and 4 new log files demonstrating all three models pass when think is left at default.
This commit is contained in:
+25
-4
@@ -25,6 +25,8 @@ Get those right and Gemma 4 just works. Get them wrong and you get a generic cha
|
||||
|
||||
## Mandatory Ollama Settings
|
||||
|
||||
### For single-turn pipelines (AI_Visualizer shape)
|
||||
|
||||
Every Gemma 4 call MUST include:
|
||||
|
||||
```json
|
||||
@@ -38,12 +40,31 @@ Every Gemma 4 call MUST include:
|
||||
```
|
||||
|
||||
**Why each one:**
|
||||
- `think: false` — Ollama 0.20+ defaults to think:true. Thinking tokens consume num_predict budget invisibly, returning empty responses. Seth has ONLY had success with thinking off.
|
||||
- `think: false` — Ollama 0.20+ defaults to think:true. In single-turn JSON pipelines, thinking tokens consume num_predict budget invisibly, returning empty responses.
|
||||
- `num_ctx: 4096+` — Ollama defaults to 2048. Your system prompt alone might exceed that.
|
||||
- `num_predict: 2048+` — Ollama defaults to 128. Any structured output gets truncated.
|
||||
|
||||
Scale these to your task. The values above are safe minimums, not recommendations.
|
||||
|
||||
### For multi-turn tool-calling agents (Simon / CLI-coding-agent shape)
|
||||
|
||||
**Do NOT set `think: false`.** Leave it unset (Ollama default) or `true`.
|
||||
|
||||
```json
|
||||
{
|
||||
"options": {
|
||||
"num_ctx": 32768,
|
||||
"num_predict": 4096
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Verified 2026-04-18 that `think: false` silently breaks `gemma4:26b` in multi-turn
|
||||
tool-calling loops — model silent-stops with `eval_count=4` at tool-decision turns.
|
||||
31B Dense and Qwen3-Coder tolerate the flag; 26B MoE does not. See `GOTCHAS.md`
|
||||
§ "`think: false` Kills Gemma 4 26B in Multi-Turn Tool-Calling Loops" and
|
||||
`docs/reference/bakeoff-2026-04-18.md` § "Round 3".
|
||||
|
||||
## System Prompt Template
|
||||
|
||||
```
|
||||
@@ -176,12 +197,12 @@ Vision is on ALL Gemma 4 variants (E2B, E4B, 26B, 31B). Audio is E-series only.
|
||||
| Maximum quality (single-model GPU) | `gemma4:31b-it-q4_K_M` | Dense 31B, sharpest but 5x slower, more VRAM pressure |
|
||||
| Rapid prototyping / testing | `gemma4:26b` | Fast enough for interactive dev |
|
||||
| Retrieval / embeddings | `embeddinggemma` (308M, separate model) | Gemma 4 has no embedding mode; use the sibling |
|
||||
| CLI coding agent (openclaw / open code / pi / hermes / aider) | `gemma4:31b-it-q4_K_M` (robust), or `gemma4:26b` if agent truncates tool responses ≤1200 chars | 2026-04-18 bakeoff on 3090 Ti: **31B clean default**; **26B silent-stops at iter 6 unless tool responses are capped — then it's the fastest passing config (8.4s, 8 iters)**. Production CLI agents typically truncate by default, so 26B may just work. See `CORPUS_cli_coding_agent.md` + `docs/reference/bakeoff-2026-04-18.md` |
|
||||
| CLI coding agent (openclaw / open code / pi / hermes / aider) | `gemma4:26b` (fastest) or `gemma4:31b-it-q4_K_M` (more headroom), either works — **just do not set `think: false` in the payload** | 2026-04-18 bakeoff on 3090 Ti: all three models (including Qwen3-Coder 30B) pass the same task in 8-14 iters. The only real gotcha is `think: false` silently breaks 26B in multi-turn loops. See `CORPUS_cli_coding_agent.md` + `docs/reference/bakeoff-2026-04-18.md` |
|
||||
|
||||
## Anti-Patterns
|
||||
|
||||
1. **Don't use `format: "json"`** — infinite loops on nested schemas
|
||||
2. **Don't leave `think` at default** — eats your output budget silently
|
||||
2. **For single-turn pipelines, don't leave `think` at default** — eats your output budget silently. **For multi-turn tool-calling agents, don't SET `think: false`** — silent-stops 26B. See the two "Mandatory Ollama Settings" sections above.
|
||||
3. **Don't leave `num_predict` at default** — 128 tokens is nothing
|
||||
4. **Don't leave `num_ctx` at default** — 2048 truncates most prompts
|
||||
5. **Don't ask for huge JSON in one call** — break into sequential calls
|
||||
@@ -191,7 +212,7 @@ Vision is on ALL Gemma 4 variants (E2B, E4B, 26B, 31B). Audio is E-series only.
|
||||
|
||||
## Quick-Start Checklist
|
||||
|
||||
- [ ] Set `think: false`
|
||||
- [ ] Set `think: false` **for single-turn pipelines only**. Leave unset for multi-turn tool-calling agents (silent-stops 26B).
|
||||
- [ ] Set `num_predict` >= 512 (2048+ for JSON output)
|
||||
- [ ] Set `num_ctx` >= 4096 (scale to your prompt size)
|
||||
- [ ] Write explicit system prompt with identity + boundaries + output format
|
||||
|
||||
Reference in New Issue
Block a user