Seth/gemma4-research
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

feat: round-2 bakeoff — 26b silent-stop is tool-response context size

Browse Source 
Round 2 tested the hypothesis that 26B's silent-stop was about
write_file argument size. Result: refuted.

- Patch-mode (apply_patch instead of write_file): 26B fails identically
  at iter 6. Tool-arg size is not the cause.
- Truncation sweep on tool responses reveals the real trigger: cap at
  800 or 1200 chars → 26B PASSES (1200-cap is 8.4s, fastest of any run).
  Cap at 1600, 2000, or unlimited → 26B silent-stops with eval=4.

Revised understanding: 26B silent-stops when cumulative tool-response
context crosses a shape threshold around 1200-1600 chars per response.
Not a tool-arg bug, not a raw code-gen bug — 26B emits correct code
fine in both one-shot and short-context settings.

Production CLI agents (openclaw, open code, aider) typically truncate
tool responses by default, so this failure may not surface in them.
Custom harnesses should cap ≤1200 chars per tool response when
targeting the 26B MoE.

Updates GOTCHAS (rewritten entry with the truncation sweep table),
SYNTHESIS model-selection row, CORPUS_cli_coding_agent.md pointer,
docs/reference/bakeoff-2026-04-18.md with full Round 2 methodology
and data.

Adds harness_patch.py (apply_patch edit tool), harness_patch_truncated.py
(env-configurable TOOL_RESULT_CAP), all 7 run logs, and a
.secrets.baseline for detect-secrets false positives on JSON timestamps.
This commit is contained in:
Mortdecai
2026-04-18 13:40:18 -04:00
parent a945207aab
commit 7f806e0b92
15 changed files with 16481 additions and 32 deletions
Download Patch File Download Diff File Expand all files Collapse all files
SYNTHESIS.md
+1 -1
View File
@@ -176,7 +176,7 @@ Vision is on ALL Gemma 4 variants (E2B, E4B, 26B, 31B). Audio is E-series only.
| Maximum quality (single-model GPU) | `gemma4:31b-it-q4_K_M` | Dense 31B, sharpest but 5x slower, more VRAM pressure |
| Rapid prototyping / testing | `gemma4:26b` | Fast enough for interactive dev |
| Retrieval / embeddings | `embeddinggemma` (308M, separate model) | Gemma 4 has no embedding mode; use the sibling |
| CLI coding agent (openclaw / open code / pi / hermes / aider) | `gemma4:31b-it-q4_K_M` (verified), fallback `qwen3-coder:30b` | 2026-04-18 bakeoff on 3090 Ti: 31B passes cleanly (8 iters, 1 write), Qwen3-Coder passes verbosely (15 iters), **26B reproducibly silent-stops at the write_file tool call** — see `CORPUS_cli_coding_agent.md` and `docs/reference/bakeoff-2026-04-18.md` |
| CLI coding agent (openclaw / open code / pi / hermes / aider) | `gemma4:31b-it-q4_K_M` (robust), or `gemma4:26b` if agent truncates tool responses ≤1200 chars | 2026-04-18 bakeoff on 3090 Ti: **31B clean default**; **26B silent-stops at iter 6 unless tool responses are capped — then it's the fastest passing config (8.4s, 8 iters)**. Production CLI agents typically truncate by default, so 26B may just work. See `CORPUS_cli_coding_agent.md` + `docs/reference/bakeoff-2026-04-18.md` |
## Anti-Patterns