Seth asked "was this with think=false?" Yes — and that was the only
question that mattered. Everything I concluded in round 1 and round 2
was wrong.
Actual cause, isolated in round 3:
- At identical message state, gemma4:26b with think=false returns
eval=4 (silent stop); with think unset or think=true, returns
eval=165 and emits the correct tool call.
- Original round-1 write_file harness + think unset: 26B passes in
8 iters, 20s. No mitigations needed.
- 31B dense and qwen3-coder:30b tolerate think=false; 26B MoE does not.
Red herrings (kept on-record in the bakeoff doc, not silently erased):
- Round 1: "write_file tool-call argument size" — wrong
- Round 2a: refuted the arg-size theory but for the wrong reason
(still failed because think=false was still set)
- Round 2b: "cumulative tool-response context size" — truncating
did make 26B pass, but by coincidence. Shorter context at the
decision turn dodged the think=false side effect.
Why the existing "always think:false" guidance was misleading:
it was derived from AI_Visualizer (single-turn JSON pipelines) where
thinking tokens do eat num_predict invisibly. In multi-turn
tool-calling agents the channels are separate and the flag has a
different effect — catastrophic on 26B specifically.
Doc updates:
- GOTCHAS: replaced the 26B entry with the actual cause; scoped the
original "Thinking Mode Eats Context" entry to single-turn pipelines
- SYNTHESIS: split the "Mandatory Ollama Settings" block into
single-turn vs multi-turn variants; updated anti-patterns and
quick-start checklist
- CORPUS_cli_coding_agent.md: revised pointer and config template
- docs/reference/bakeoff-2026-04-18.md: added Round 3 section with
the correction notice at the top of the file and full diagnostic
methodology
New artifacts: harness_no_think_flag.py, harness_write_no_think.py,
and 4 new log files demonstrating all three models pass when think
is left at default.
Round 2 tested the hypothesis that 26B's silent-stop was about
write_file argument size. Result: refuted.
- Patch-mode (apply_patch instead of write_file): 26B fails identically
at iter 6. Tool-arg size is not the cause.
- Truncation sweep on tool responses reveals the real trigger: cap at
800 or 1200 chars → 26B PASSES (1200-cap is 8.4s, fastest of any run).
Cap at 1600, 2000, or unlimited → 26B silent-stops with eval=4.
Revised understanding: 26B silent-stops when cumulative tool-response
context crosses a shape threshold around 1200-1600 chars per response.
Not a tool-arg bug, not a raw code-gen bug — 26B emits correct code
fine in both one-shot and short-context settings.
Production CLI agents (openclaw, open code, aider) typically truncate
tool responses by default, so this failure may not surface in them.
Custom harnesses should cap ≤1200 chars per tool response when
targeting the 26B MoE.
Updates GOTCHAS (rewritten entry with the truncation sweep table),
SYNTHESIS model-selection row, CORPUS_cli_coding_agent.md pointer,
docs/reference/bakeoff-2026-04-18.md with full Round 2 methodology
and data.
Adds harness_patch.py (apply_patch edit tool), harness_patch_truncated.py
(env-configurable TOOL_RESULT_CAP), all 7 run logs, and a
.secrets.baseline for detect-secrets false positives on JSON timestamps.
Ran minimal agent loop (Ollama /api/chat + read_file/write_file/run_bash) on
steel141 3090 Ti against 3 models on a broken-median-function task:
- gemma4:31b-it-q4_K_M: PASS (8 iters, 1 write, 44s) — textbook trace
- qwen3-coder:30b: PASS (15 iters, 1 write, 22s) — correct but chatty
- gemma4:26b: FAIL (6 iters, 0 writes) — silently stops with eval=4
after reading source. Reproduced on second run. One-shot probe
confirms 26b CAN produce the correct fix — failure is specifically
at the write_file tool-call argument boundary.
Updates GOTCHAS with a new HIGH-severity entry, SYNTHESIS model-selection
table, CORPUS_cli_coding_agent.md empirical-follow-up pointer, and adds
docs/reference/bakeoff-2026-04-18.md with the full writeup.
Mortdecai