fix: walk back round-1/2 conclusions — the cause was think=false all along
Seth asked "was this with think=false?" Yes — and that was the only question that mattered. Everything I concluded in round 1 and round 2 was wrong. Actual cause, isolated in round 3: - At identical message state, gemma4:26b with think=false returns eval=4 (silent stop); with think unset or think=true, returns eval=165 and emits the correct tool call. - Original round-1 write_file harness + think unset: 26B passes in 8 iters, 20s. No mitigations needed. - 31B dense and qwen3-coder:30b tolerate think=false; 26B MoE does not. Red herrings (kept on-record in the bakeoff doc, not silently erased): - Round 1: "write_file tool-call argument size" — wrong - Round 2a: refuted the arg-size theory but for the wrong reason (still failed because think=false was still set) - Round 2b: "cumulative tool-response context size" — truncating did make 26B pass, but by coincidence. Shorter context at the decision turn dodged the think=false side effect. Why the existing "always think:false" guidance was misleading: it was derived from AI_Visualizer (single-turn JSON pipelines) where thinking tokens do eat num_predict invisibly. In multi-turn tool-calling agents the channels are separate and the flag has a different effect — catastrophic on 26B specifically. Doc updates: - GOTCHAS: replaced the 26B entry with the actual cause; scoped the original "Thinking Mode Eats Context" entry to single-turn pipelines - SYNTHESIS: split the "Mandatory Ollama Settings" block into single-turn vs multi-turn variants; updated anti-patterns and quick-start checklist - CORPUS_cli_coding_agent.md: revised pointer and config template - docs/reference/bakeoff-2026-04-18.md: added Round 3 section with the correction notice at the top of the file and full diagnostic methodology New artifacts: harness_no_think_flag.py, harness_write_no_think.py, and 4 new log files demonstrating all three models pass when think is left at default.
This commit is contained in:
Mortdecai
+51
-45
@@ -3,17 +3,25 @@
|
||||
> Derived from Seth's production implementations (Simon, AI_Visualizer)
|
||||
> and community reports. These are hard-won lessons.
|
||||
|
||||
## CRITICAL: Thinking Mode Eats Context
|
||||
## CRITICAL: Thinking Mode Eats Context (single-turn pipelines only)
|
||||
|
||||
**Severity: HIGH — causes silent failures**
|
||||
**Severity: HIGH — causes silent failures in single-turn `/api/generate` workloads**
|
||||
|
||||
Gemma 4 in Ollama 0.20+ defaults to `think: true`. When enabled:
|
||||
> **Scope update (2026-04-18):** This guidance applies to **single-turn JSON
|
||||
> generation pipelines** (the AI_Visualizer shape: one call → one structured
|
||||
> response). For **multi-turn tool-calling agents**, the opposite is true on
|
||||
> `gemma4:26b` — see § "`think: false` Kills Gemma 4 26B in Multi-Turn
|
||||
> Tool-Calling Loops" above. Don't copy this fix to an agent harness without
|
||||
> testing.
|
||||
|
||||
Gemma 4 in Ollama 0.20+ defaults to `think: true`. When enabled in a single-turn
|
||||
JSON pipeline:
|
||||
- Thinking tokens go into a hidden `thinking` field, NOT `response`
|
||||
- If `num_predict` is limited, thinking consumes the entire budget
|
||||
- `response` comes back **empty** — no error, just silence
|
||||
- On evaluative tasks, thinking inflates scores (31B scored a known-bad image 9/10 with thinking vs 7/10 without)
|
||||
|
||||
**Fix:** Always pass `think: false` in the Ollama payload. Seth has had success ONLY with thinking off.
|
||||
**Fix (for single-turn pipelines):** Always pass `think: false` in the Ollama payload.
|
||||
|
||||
```json
|
||||
{
|
||||
@@ -23,6 +31,9 @@ Gemma 4 in Ollama 0.20+ defaults to `think: true`. When enabled:
|
||||
}
|
||||
```
|
||||
|
||||
**Do not blindly carry this to multi-turn tool-calling agents** — verified
|
||||
2026-04-18 that it silent-stops 26B specifically in that context.
|
||||
|
||||
## CRITICAL: format=json Causes Infinite Loops
|
||||
|
||||
**Severity: HIGH — hangs indefinitely**
|
||||
@@ -64,61 +75,56 @@ Ollama defaults `num_predict` to 128 tokens. Almost any useful Gemma 4 output ex
|
||||
|
||||
**Fix:** Always set `num_predict` explicitly. Minimum recommended: 512. For JSON output: 2048+.
|
||||
|
||||
## HIGH: 26B Silent-Stops When Tool Responses Accumulate (reproducible)
|
||||
## HIGH: `think: false` Kills Gemma 4 26B in Multi-Turn Tool-Calling Loops
|
||||
|
||||
**Severity: HIGH — silent agent-loop failure. Mitigatable.**
|
||||
**Severity: HIGH — silent agent-loop failure. Setting is what the old guidance said to do.**
|
||||
|
||||
Reproduced on 2026-04-18 against `gemma4:26b` via Ollama 0.20.4 on a 3090 Ti
|
||||
(steel141). Agent harness looped through `read_file` / `(write_file or apply_patch)` / `run_bash`
|
||||
tools to fix a failing Python test.
|
||||
(steel141). Contradicts the older "always think:false" guidance (see § "Thinking
|
||||
Mode Eats Context" below — now scoped to single-turn pipelines only).
|
||||
|
||||
### The observation
|
||||
|
||||
26B silent-stops (empty content, no tool calls, `eval_count=4`) at the
|
||||
decision-to-edit turn, **regardless of which edit tool is offered** — tested with
|
||||
both `write_file(path, full_content)` and `apply_patch(path, old, new)`.
|
||||
Initial hypothesis (long tool-call argument) was **refuted**.
|
||||
At identical message state with all else equal:
|
||||
|
||||
### The actual trigger: cumulative tool-response context shape
|
||||
|
||||
A sweep with progressive truncation caps on tool responses (`TOOL_RESULT_CAP`):
|
||||
|
||||
| Cap (chars) | Result | Halt eval_count |
|
||||
| `think` setting | `eval_count` on decision turn | Agent behavior |
|
||||
|---|---|---|
|
||||
| 800 | PASS | 24 (continues, hits iteration cap) |
|
||||
| 1200 | **PASS** — **fastest of any run (8.4s)** | 27 (clean summary) |
|
||||
| 1600 | FAIL | **4** (silent stop) |
|
||||
| 2000 | FAIL | **4** (silent stop) |
|
||||
| unlimited | FAIL | **4** (silent stop) |
|
||||
| `false` | **4** (silent stop, no content, no tool_calls) | Fails — zero edits emitted |
|
||||
| unset (Ollama default) | 165 | Passes — emits correct edit |
|
||||
| `true` | 165 | Passes — emits correct edit |
|
||||
|
||||
Sharp transition between 1200 and 1600 chars-per-response. Below the line, 26B
|
||||
emits correct code (eval_count ~165 on the patch turn). Above, it silent-stops.
|
||||
Exact mechanism unproven (could be MoE expert routing, chat-template edge case,
|
||||
or something else). **Actionable:** cap tool responses ≤1200 chars.
|
||||
26B passes the task in 8 iterations / 12-20s on the same harness the moment
|
||||
the `think` key is removed from the Ollama payload. `write_file` vs
|
||||
`apply_patch` doesn't matter. Tool-response size doesn't matter.
|
||||
|
||||
### What's NOT at fault
|
||||
### What I initially got wrong
|
||||
|
||||
- **Not the edit tool surface** — `write_file` and `apply_patch` both trigger it
|
||||
- **Not raw code generation** — a one-shot direct prompt asking 26B to fix the
|
||||
same function returned clean correct code (eval=81)
|
||||
- **Not total context size alone** — the 800-cap run continued past 3741 prompt
|
||||
tokens. Failing runs halt at ~2070-2150 tokens but the 1200-cap run crossed
|
||||
the same range and kept going
|
||||
- **Not a Gemma-4-family issue** — `gemma4:31b-it-q4_K_M` on identical harness
|
||||
handles full-size tool responses cleanly (eval=330 on the write turn)
|
||||
The 2026-04-18 bakeoff went through two wrong hypotheses before Seth asked
|
||||
"was this with think=false?" The failed-and-corrected path:
|
||||
|
||||
1. **"Long `write_file` argument breaks 26B"** — wrong. `apply_patch` also failed.
|
||||
2. **"Large tool-response context breaks 26B"** — wrong. Truncation *did* make
|
||||
26B pass (800/1200-char caps), but that's because shorter context dodged
|
||||
the `think: false` side effect by coincidence of state at the decision turn.
|
||||
3. **Actual cause:** `think: false` alters the decoding path in a way that makes
|
||||
the 26B MoE (3.8B active params, 8-of-128 expert routing) emit near-immediate
|
||||
EOS at tool-decision turns. 31B Dense and Qwen3-Coder are robust to the
|
||||
flag; 26B specifically is not.
|
||||
|
||||
See `docs/reference/bakeoff-2026-04-18.md` § "Round 3" for full traces and the
|
||||
diagnostic that isolated the flag.
|
||||
|
||||
### Fix
|
||||
|
||||
- **For 26B in an agent loop, cap tool responses ≤1200 chars.** 800 is safer;
|
||||
this is where every production CLI agent (openclaw / open code / aider /
|
||||
cline) already lives by default, so the issue may not surface in those
|
||||
frameworks.
|
||||
- **For raw pytest output specifically**, use `pytest -x --tb=line` or a custom
|
||||
formatter to shrink per-test output to a few lines.
|
||||
- **Alternative:** use `gemma4:31b-it-q4_K_M` — same harness, no mitigation,
|
||||
just works. Trade: ~5× slower than 26B when 26B cooperates.
|
||||
- See `docs/reference/bakeoff-2026-04-18.md` (Round 2) for full traces and the
|
||||
truncation sweep methodology.
|
||||
- **For multi-turn tool-calling agents, do NOT set `think: false`.** Leave it
|
||||
unset (Ollama default) or `true`.
|
||||
- **If your agent accumulates `thinking` field content**, prune old thinking
|
||||
blobs from message history to control context growth.
|
||||
- **For single-turn JSON pipelines** (the AI_Visualizer shape), the original
|
||||
"always think:false" guidance still applies — see § "Thinking Mode Eats
|
||||
Context" below.
|
||||
- 31B Dense and Qwen3-Coder work fine either way — this gotcha is 26B-specific
|
||||
on this Ollama version.
|
||||
|
||||
## MEDIUM: Weak at Long/Nested JSON
|
||||
|
||||
|
||||
Reference in New Issue
Block a user