fix: walk back round-1/2 conclusions — the cause was think=false all along

Seth asked "was this with think=false?" Yes — and that was the only
question that mattered. Everything I concluded in round 1 and round 2
was wrong.

Actual cause, isolated in round 3:
- At identical message state, gemma4:26b with think=false returns
  eval=4 (silent stop); with think unset or think=true, returns
  eval=165 and emits the correct tool call.
- Original round-1 write_file harness + think unset: 26B passes in
  8 iters, 20s. No mitigations needed.
- 31B dense and qwen3-coder:30b tolerate think=false; 26B MoE does not.

Red herrings (kept on-record in the bakeoff doc, not silently erased):
- Round 1: "write_file tool-call argument size" — wrong
- Round 2a: refuted the arg-size theory but for the wrong reason
  (still failed because think=false was still set)
- Round 2b: "cumulative tool-response context size" — truncating
  did make 26B pass, but by coincidence. Shorter context at the
  decision turn dodged the think=false side effect.

Why the existing "always think:false" guidance was misleading:
it was derived from AI_Visualizer (single-turn JSON pipelines) where
thinking tokens do eat num_predict invisibly. In multi-turn
tool-calling agents the channels are separate and the flag has a
different effect — catastrophic on 26B specifically.

Doc updates:
- GOTCHAS: replaced the 26B entry with the actual cause; scoped the
  original "Thinking Mode Eats Context" entry to single-turn pipelines
- SYNTHESIS: split the "Mandatory Ollama Settings" block into
  single-turn vs multi-turn variants; updated anti-patterns and
  quick-start checklist
- CORPUS_cli_coding_agent.md: revised pointer and config template
- docs/reference/bakeoff-2026-04-18.md: added Round 3 section with
  the correction notice at the top of the file and full diagnostic
  methodology

New artifacts: harness_no_think_flag.py, harness_write_no_think.py,
and 4 new log files demonstrating all three models pass when think
is left at default.
This commit is contained in:
Mortdecai
2026-04-18 18:14:05 -04:00
parent 7f806e0b92
commit c61394923c
11 changed files with 1132 additions and 61 deletions
+25 -4
View File
@@ -25,6 +25,8 @@ Get those right and Gemma 4 just works. Get them wrong and you get a generic cha
## Mandatory Ollama Settings
### For single-turn pipelines (AI_Visualizer shape)
Every Gemma 4 call MUST include:
```json
@@ -38,12 +40,31 @@ Every Gemma 4 call MUST include:
```
**Why each one:**
- `think: false` — Ollama 0.20+ defaults to think:true. Thinking tokens consume num_predict budget invisibly, returning empty responses. Seth has ONLY had success with thinking off.
- `think: false` — Ollama 0.20+ defaults to think:true. In single-turn JSON pipelines, thinking tokens consume num_predict budget invisibly, returning empty responses.
- `num_ctx: 4096+` — Ollama defaults to 2048. Your system prompt alone might exceed that.
- `num_predict: 2048+` — Ollama defaults to 128. Any structured output gets truncated.
Scale these to your task. The values above are safe minimums, not recommendations.
### For multi-turn tool-calling agents (Simon / CLI-coding-agent shape)
**Do NOT set `think: false`.** Leave it unset (Ollama default) or `true`.
```json
{
"options": {
"num_ctx": 32768,
"num_predict": 4096
}
}
```
Verified 2026-04-18 that `think: false` silently breaks `gemma4:26b` in multi-turn
tool-calling loops — model silent-stops with `eval_count=4` at tool-decision turns.
31B Dense and Qwen3-Coder tolerate the flag; 26B MoE does not. See `GOTCHAS.md`
§ "`think: false` Kills Gemma 4 26B in Multi-Turn Tool-Calling Loops" and
`docs/reference/bakeoff-2026-04-18.md` § "Round 3".
## System Prompt Template
```
@@ -176,12 +197,12 @@ Vision is on ALL Gemma 4 variants (E2B, E4B, 26B, 31B). Audio is E-series only.
| Maximum quality (single-model GPU) | `gemma4:31b-it-q4_K_M` | Dense 31B, sharpest but 5x slower, more VRAM pressure |
| Rapid prototyping / testing | `gemma4:26b` | Fast enough for interactive dev |
| Retrieval / embeddings | `embeddinggemma` (308M, separate model) | Gemma 4 has no embedding mode; use the sibling |
| CLI coding agent (openclaw / open code / pi / hermes / aider) | `gemma4:31b-it-q4_K_M` (robust), or `gemma4:26b` if agent truncates tool responses ≤1200 chars | 2026-04-18 bakeoff on 3090 Ti: **31B clean default**; **26B silent-stops at iter 6 unless tool responses are capped — then it's the fastest passing config (8.4s, 8 iters)**. Production CLI agents typically truncate by default, so 26B may just work. See `CORPUS_cli_coding_agent.md` + `docs/reference/bakeoff-2026-04-18.md` |
| CLI coding agent (openclaw / open code / pi / hermes / aider) | `gemma4:26b` (fastest) or `gemma4:31b-it-q4_K_M` (more headroom), either works — **just do not set `think: false` in the payload** | 2026-04-18 bakeoff on 3090 Ti: all three models (including Qwen3-Coder 30B) pass the same task in 8-14 iters. The only real gotcha is `think: false` silently breaks 26B in multi-turn loops. See `CORPUS_cli_coding_agent.md` + `docs/reference/bakeoff-2026-04-18.md` |
## Anti-Patterns
1. **Don't use `format: "json"`** — infinite loops on nested schemas
2. **Don't leave `think` at default** — eats your output budget silently
2. **For single-turn pipelines, don't leave `think` at default** — eats your output budget silently. **For multi-turn tool-calling agents, don't SET `think: false`** — silent-stops 26B. See the two "Mandatory Ollama Settings" sections above.
3. **Don't leave `num_predict` at default** — 128 tokens is nothing
4. **Don't leave `num_ctx` at default** — 2048 truncates most prompts
5. **Don't ask for huge JSON in one call** — break into sequential calls
@@ -191,7 +212,7 @@ Vision is on ALL Gemma 4 variants (E2B, E4B, 26B, 31B). Audio is E-series only.
## Quick-Start Checklist
- [ ] Set `think: false`
- [ ] Set `think: false` **for single-turn pipelines only**. Leave unset for multi-turn tool-calling agents (silent-stops 26B).
- [ ] Set `num_predict` >= 512 (2048+ for JSON output)
- [ ] Set `num_ctx` >= 4096 (scale to your prompt size)
- [ ] Write explicit system prompt with identity + boundaries + output format