fix: walk back round-1/2 conclusions — the cause was think=false all along
Seth asked "was this with think=false?" Yes — and that was the only question that mattered. Everything I concluded in round 1 and round 2 was wrong. Actual cause, isolated in round 3: - At identical message state, gemma4:26b with think=false returns eval=4 (silent stop); with think unset or think=true, returns eval=165 and emits the correct tool call. - Original round-1 write_file harness + think unset: 26B passes in 8 iters, 20s. No mitigations needed. - 31B dense and qwen3-coder:30b tolerate think=false; 26B MoE does not. Red herrings (kept on-record in the bakeoff doc, not silently erased): - Round 1: "write_file tool-call argument size" — wrong - Round 2a: refuted the arg-size theory but for the wrong reason (still failed because think=false was still set) - Round 2b: "cumulative tool-response context size" — truncating did make 26B pass, but by coincidence. Shorter context at the decision turn dodged the think=false side effect. Why the existing "always think:false" guidance was misleading: it was derived from AI_Visualizer (single-turn JSON pipelines) where thinking tokens do eat num_predict invisibly. In multi-turn tool-calling agents the channels are separate and the flag has a different effect — catastrophic on 26B specifically. Doc updates: - GOTCHAS: replaced the 26B entry with the actual cause; scoped the original "Thinking Mode Eats Context" entry to single-turn pipelines - SYNTHESIS: split the "Mandatory Ollama Settings" block into single-turn vs multi-turn variants; updated anti-patterns and quick-start checklist - CORPUS_cli_coding_agent.md: revised pointer and config template - docs/reference/bakeoff-2026-04-18.md: added Round 3 section with the correction notice at the top of the file and full diagnostic methodology New artifacts: harness_no_think_flag.py, harness_write_no_think.py, and 4 new log files demonstrating all three models pass when think is left at default.
This commit is contained in:
@@ -0,0 +1,83 @@
|
||||
{
|
||||
"model": "gemma4:26b",
|
||||
"edit_tool": "write_file",
|
||||
"think_setting": "unset (default)",
|
||||
"turns": [
|
||||
{
|
||||
"iteration": 1,
|
||||
"elapsed_s": 8.52,
|
||||
"content": "",
|
||||
"prompt_eval_count": 369,
|
||||
"eval_count": 55,
|
||||
"tool_calls_count": 1
|
||||
},
|
||||
{
|
||||
"iteration": 2,
|
||||
"elapsed_s": 1.07,
|
||||
"content": "",
|
||||
"prompt_eval_count": 510,
|
||||
"eval_count": 46,
|
||||
"tool_calls_count": 1
|
||||
},
|
||||
{
|
||||
"iteration": 3,
|
||||
"elapsed_s": 0.75,
|
||||
"content": "",
|
||||
"prompt_eval_count": 691,
|
||||
"eval_count": 19,
|
||||
"tool_calls_count": 1
|
||||
},
|
||||
{
|
||||
"iteration": 4,
|
||||
"elapsed_s": 1.05,
|
||||
"content": "",
|
||||
"prompt_eval_count": 1140,
|
||||
"eval_count": 27,
|
||||
"tool_calls_count": 1
|
||||
},
|
||||
{
|
||||
"iteration": 5,
|
||||
"elapsed_s": 1.19,
|
||||
"content": "",
|
||||
"prompt_eval_count": 1786,
|
||||
"eval_count": 23,
|
||||
"tool_calls_count": 1
|
||||
},
|
||||
{
|
||||
"iteration": 6,
|
||||
"elapsed_s": 4.87,
|
||||
"content": "",
|
||||
"prompt_eval_count": 2075,
|
||||
"eval_count": 330,
|
||||
"tool_calls_count": 1
|
||||
},
|
||||
{
|
||||
"iteration": 7,
|
||||
"elapsed_s": 0.99,
|
||||
"content": "",
|
||||
"prompt_eval_count": 2422,
|
||||
"eval_count": 26,
|
||||
"tool_calls_count": 1
|
||||
},
|
||||
{
|
||||
"iteration": 8,
|
||||
"elapsed_s": 0.96,
|
||||
"content": "The `median` function was updated to correctly average the two middle elements when the input list has an even number of elements.",
|
||||
"prompt_eval_count": 2641,
|
||||
"eval_count": 30,
|
||||
"tool_calls_count": 0
|
||||
}
|
||||
],
|
||||
"started_at": 1776550159.4912357,
|
||||
"final": {
|
||||
"halt_reason": "no_tool_calls",
|
||||
"tests_pass": true,
|
||||
"iterations_used": 8,
|
||||
"tool_call_counts": {
|
||||
"read_file": 2,
|
||||
"write_file": 1,
|
||||
"run_bash": 4
|
||||
},
|
||||
"wall_clock_s": 20.58
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user