feat: round-2 bakeoff — 26b silent-stop is tool-response context size
Round 2 tested the hypothesis that 26B's silent-stop was about write_file argument size. Result: refuted. - Patch-mode (apply_patch instead of write_file): 26B fails identically at iter 6. Tool-arg size is not the cause. - Truncation sweep on tool responses reveals the real trigger: cap at 800 or 1200 chars → 26B PASSES (1200-cap is 8.4s, fastest of any run). Cap at 1600, 2000, or unlimited → 26B silent-stops with eval=4. Revised understanding: 26B silent-stops when cumulative tool-response context crosses a shape threshold around 1200-1600 chars per response. Not a tool-arg bug, not a raw code-gen bug — 26B emits correct code fine in both one-shot and short-context settings. Production CLI agents (openclaw, open code, aider) typically truncate tool responses by default, so this failure may not surface in them. Custom harnesses should cap ≤1200 chars per tool response when targeting the 26B MoE. Updates GOTCHAS (rewritten entry with the truncation sweep table), SYNTHESIS model-selection row, CORPUS_cli_coding_agent.md pointer, docs/reference/bakeoff-2026-04-18.md with full Round 2 methodology and data. Adds harness_patch.py (apply_patch edit tool), harness_patch_truncated.py (env-configurable TOOL_RESULT_CAP), all 7 run logs, and a .secrets.baseline for detect-secrets false positives on JSON timestamps.
This commit is contained in:
@@ -6,13 +6,16 @@
|
||||
> `IMPLEMENTATIONS.md` chat-agent patterns (Simon) and pipeline patterns
|
||||
> (AI_Visualizer).
|
||||
|
||||
> **Empirical follow-up:** `docs/reference/bakeoff-2026-04-18.md` — real runs of
|
||||
> `gemma4:26b`, `gemma4:31b-it-q4_K_M`, and `qwen3-coder:30b` against a custom
|
||||
> minimal CLI-agent harness on a fix-the-median-bug task. Key findings:
|
||||
> **31B clean (8 iters, 1 write), Qwen3-Coder correct but chatty (15 iters),
|
||||
> 26B reproducibly silent-stops at the `write_file` tool call boundary** even
|
||||
> though it can produce the fix in a direct one-shot call. Read when: scoping
|
||||
> which model to point an agent at, or hitting an unexpected tool-call halt.
|
||||
> **Empirical follow-up:** `docs/reference/bakeoff-2026-04-18.md` — 2 rounds of
|
||||
> runs against a custom minimal CLI-agent harness on a fix-the-median-bug task.
|
||||
> **Round 1:** 31B clean (8 iters), Qwen3-Coder correct but chatty (15 iters),
|
||||
> 26B silently quits with zero edits. **Round 2 (diagnostic):** the 26B failure
|
||||
> is NOT about edit-tool-argument size — it's about **cumulative tool-response
|
||||
> context shape**. Capping tool responses ≤1200 chars makes 26B pass cleanly
|
||||
> *and* in the fastest wall time of any run (8.4s). Most production CLI agents
|
||||
> already truncate tool responses, so the issue may be invisible in them.
|
||||
> Read when: scoping which model to point an agent at, hitting an unexpected
|
||||
> tool-call halt, or writing a custom harness targeting the 26B MoE.
|
||||
|
||||
## TL;DR
|
||||
|
||||
|
||||
Reference in New Issue
Block a user