Round 2 tested the hypothesis that 26B's silent-stop was about
write_file argument size. Result: refuted.
- Patch-mode (apply_patch instead of write_file): 26B fails identically
at iter 6. Tool-arg size is not the cause.
- Truncation sweep on tool responses reveals the real trigger: cap at
800 or 1200 chars → 26B PASSES (1200-cap is 8.4s, fastest of any run).
Cap at 1600, 2000, or unlimited → 26B silent-stops with eval=4.
Revised understanding: 26B silent-stops when cumulative tool-response
context crosses a shape threshold around 1200-1600 chars per response.
Not a tool-arg bug, not a raw code-gen bug — 26B emits correct code
fine in both one-shot and short-context settings.
Production CLI agents (openclaw, open code, aider) typically truncate
tool responses by default, so this failure may not surface in them.
Custom harnesses should cap ≤1200 chars per tool response when
targeting the 26B MoE.
Updates GOTCHAS (rewritten entry with the truncation sweep table),
SYNTHESIS model-selection row, CORPUS_cli_coding_agent.md pointer,
docs/reference/bakeoff-2026-04-18.md with full Round 2 methodology
and data.
Adds harness_patch.py (apply_patch edit tool), harness_patch_truncated.py
(env-configurable TOOL_RESULT_CAP), all 7 run logs, and a
.secrets.baseline for detect-secrets false positives on JSON timestamps.
Ran minimal agent loop (Ollama /api/chat + read_file/write_file/run_bash) on
steel141 3090 Ti against 3 models on a broken-median-function task:
- gemma4:31b-it-q4_K_M: PASS (8 iters, 1 write, 44s) — textbook trace
- qwen3-coder:30b: PASS (15 iters, 1 write, 22s) — correct but chatty
- gemma4:26b: FAIL (6 iters, 0 writes) — silently stops with eval=4
after reading source. Reproduced on second run. One-shot probe
confirms 26b CAN produce the correct fix — failure is specifically
at the write_file tool-call argument boundary.
Updates GOTCHAS with a new HIGH-severity entry, SYNTHESIS model-selection
table, CORPUS_cli_coding_agent.md empirical-follow-up pointer, and adds
docs/reference/bakeoff-2026-04-18.md with the full writeup.
Fills the gap between existing chat-agent (Simon) and pipeline
(AI_Visualizer) patterns. Covers openclaw/open code/pi/hermes first-party
agents from the HF launch blog, honest positioning vs qwen3-coder:30b,
CLI-agent-specific gotchas (safety filter on security code, long-JSON
weakness, no code exec), and a concrete homelab bakeoff plan pointed at
CT 166 openclaw2 → CT 105 Ollama on pve197.
Key research finding: Google published LiveCodeBench + Codeforces but
NOT SWE-bench or Aider polyglot. The "autonomous agents" claim is
plausible but unproven for multi-file repo-scale coding specifically.
Patches the top-level corpus docs with the 13 findings flagged during the
2026-04-18 canonical tooling research pass. tooling/README.md now marks each
finding [merged: <file>] or [flagged] for provenance.
- CORPUS_ollama_variants.md: annotate gemma4:26b as MoE (25.2B total / 3.8B
active, 8-of-128 experts + 1 shared). Note Q4_K_M inference is standard
(the "MoE quality degrades at 4-bit" caveat is training-only). Add note
that audio on E-series is NOT available via Ollama — llama.cpp mmproj
or vLLM only.
- CORPUS_capabilities.md: native system role, configurable thinking mode,
first trained tool use (vs Gemma 1/2/3 proof-of-concept), native object
detection with bbox output in 1000x1000 coords, pointer to EmbeddingGemma
for retrieval (Gemma 4 has no embedding mode).
- CORPUS_tool_calling_format.md: add Chat Template Context section
documenting the <|turn>/<turn|> asymmetric brackets (new in Gemma 4,
replaced <start_of_turn>/<end_of_turn>) plus <|think>, <|channel>,
<|image>, <|audio> tokens. Add HF transformers Alternative section
showing processor.parse_response with response_schema.
- GOTCHAS.md: add MEDIUM gotcha for abandoned google/gemma_pytorch (no
Gemma 4 support since 2025-05-30). Expand fine-tuning section with FA2/FA4
head_dim=512 break, fused LoRA kernel issues, 26B A4B training-quant
guidance, new tool-call tokens as learned embeddings.
- SYNTHESIS.md: add banner pointing to tooling/ for canonical upstream
material. Add embeddinggemma row to Model Selection table.
Also:
- Add .gitignore excluding .backup/ (local scratch per global CLAUDE.md
convention, not needed in tracked history) and __pycache__/.
- Add .claude/handoffs/2026-04-18-canonical-tooling-research.md so future
sessions can pick up cold — facts verified, open threads, what changed.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Architecture specs, benchmarks, gotchas, Ollama settings, tool calling
format, and implementation patterns from Simon and AI_Visualizer.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Mortdecai