Cross-host Gemma 4 throughput comparison across three architectures.
Harness at scripts/gpu-bakeoff/; writeup at
docs/reference/gpu-bakeoff-2026-04-20.md.
Key findings:
- RTX 3090 Ti wins decode decisively (128 tok/s on gemma4:26b MoE Q4,
~4.7× faster than gemma4:31b dense on the same card).
- AMD Strix Halo iGPU lands at ~42% of 3090 Ti decode on ~25% of the
memory bandwidth — good SIMD utilization, especially for MoE.
- V100 numbers are DEGRADED: CT 167 ai-visualizer SDXL consumes 31/32
GB of its VRAM, forcing Gemma 4 models 95% onto CPU. Isolated V100
run requires SDXL eviction — left as follow-up.
- MoE vs dense is the dominant latency factor across all GPUs: ~4 B
active params of gemma4:26b beats 31.3 B active of gemma4:31b by
the same ratio (~4.7×) on every card tested.
Methodology: 1 warmup + 3 measurement runs per (host × model ×
prompt-length), Ollama's canonical timing fields, temp=0 greedy,
num_predict=256. All three Ollama servers accessed via HTTP (Strix
via Tailscale).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three-arm harness under scripts/native-bakeoff/:
- arm A: /api/chat with JSON tools (current default)
- arm B: /api/generate raw:true with canonical HF jinja template rendered directly
- arm C: google-deepmind/gemma JAX ToolSampler (env-gated, JAX required)
Interim finding from A+B sweep on matt-strix gemma4:26b Q4: Ollama's
bidirectional JSON↔native tool-call translator is faithful. The "long"
multi-tool task produces identical behavior (7 steps / 6 tools) on both
arms. Earlier arm-B parser bug that looked like a divergence was a
harness issue: preserving the model's <|channel>thought\n<channel|>
prefix as assistant content tripped the jinja template's
tool_response-following conditional, appending a spurious <turn|>\n
that corrupted the next step's prompt. Fixed by dropping the channel
prefix on the assistant message.
Arm C left as scaffolded-but-not-run — the JAX/bf16 reference path
would answer "does the GGUF runtime diverge from DeepMind's
implementation" but requires a separate env with the `gemma` PyPI
package. Parked pending SDXL eviction or vast-h100 session.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Strip identifying details from the gemma4 correspondence test:
contact name, file paths that imply the project, and
manuscript-specific terminology that would identify the
collaborator. Technical findings about gemma4 unchanged.
NOTE: prior commit 3ceed5c contains the unredacted version
in git history. History rewrite (force-push) requires explicit
authorization.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Single-shot test against real Andy inbound + CONTEXT.md slice.
Findings: gemma4 handles state bookkeeping well (correctly diffs
Pending, honors hard rules like rejected-analogy avoidance, uses
agreed vocabulary). Fails on precision: hallucinated message ID,
invented Figure 1 axes it had no access to, drifted off voice
register without few-shot examples.
Verdict: viable for low-stakes social correspondence + first-pass
triage; disqualified from high-stakes drafting where exact IDs
or artifact references must round-trip. Hybrid pattern proposed
(gemma4 for bookkeeping, Claude for drafting).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Chains from 2026-04-18-canonical-tooling-research. Captures the
settings-guide work shipped in d9477da and the repo-convention
note (push-on-commit for ~/bin Gitea projects).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Applies SYNTHESIS.md + GOTCHAS.md findings to the OpenWebUI front-end:
per-setting reference, two baked-in Workspace Model profiles (chat +
extract), and a symptom→cause troubleshooting table. Front-loads the
`think: false` / gemma4:26b multi-turn footgun from Round 3 of the
2026-04-18 bakeoff since that is the shape OpenWebUI users will hit.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Seth's challenge: "we experienced this context eating with every
implementation that had think=true. mort-bot runs a loop. Can you do
a bake-off?"
Built a harness that replicates mort-bot's /api/chat loop verbatim
(num_ctx=8192, num_predict=2048, temperature=0.7, gemma4:26b,
STEP_BUDGET=20, exact payload shape) but with stubbed tools and a
prebuilt 15-turn fake chat history. Ran 4 tasks × 2 think settings.
Finding: on Ollama 0.20.4 the "thinking eats context" concern does
NOT reproduce. Direct evidence:
- Movies task step 2 (think=true) returned 905 chars of thinking.
- Step 3 prompt_eval_count delta: +76 tokens (think=true) vs +135
tokens (think=false). If thinking had accumulated in the prompt,
think=true would have grown by +360 tokens, not shrunk.
- Ollama's chat template strips the `thinking` field when serializing
assistant turns for subsequent prompts.
All 4 tasks × 2 settings produced identical step counts and tool
counts. Wall clocks comparable. Gemma only actually generated
thinking on 1 of 4 tasks (the one with check_sethflix verify-loop);
on the others with think=true it emitted 0 thinking tokens.
Reconciled with the earlier coding-agent bakeoff: the two findings
are orthogonal. Coding bakeoff was at num_ctx=32K with a different
harness; mort at 8K doesn't touch the silent-stop regime either way.
Seth's prior may have been correct on an older Ollama or in a
different API shape (/api/generate has its own issues) but does not
reproduce here.
Concrete recommendation: mort-bot THINK=False is defensible but not
load-bearing; THINK=True or unset-default would also work. Keep as-is
unless a different need arises.
New: docs/reference/mort-bakeoff-2026-04-18.md, scripts/mort-bakeoff/
(harness + 8 run logs). README updated with pointer.
Seth asked "was this with think=false?" Yes — and that was the only
question that mattered. Everything I concluded in round 1 and round 2
was wrong.
Actual cause, isolated in round 3:
- At identical message state, gemma4:26b with think=false returns
eval=4 (silent stop); with think unset or think=true, returns
eval=165 and emits the correct tool call.
- Original round-1 write_file harness + think unset: 26B passes in
8 iters, 20s. No mitigations needed.
- 31B dense and qwen3-coder:30b tolerate think=false; 26B MoE does not.
Red herrings (kept on-record in the bakeoff doc, not silently erased):
- Round 1: "write_file tool-call argument size" — wrong
- Round 2a: refuted the arg-size theory but for the wrong reason
(still failed because think=false was still set)
- Round 2b: "cumulative tool-response context size" — truncating
did make 26B pass, but by coincidence. Shorter context at the
decision turn dodged the think=false side effect.
Why the existing "always think:false" guidance was misleading:
it was derived from AI_Visualizer (single-turn JSON pipelines) where
thinking tokens do eat num_predict invisibly. In multi-turn
tool-calling agents the channels are separate and the flag has a
different effect — catastrophic on 26B specifically.
Doc updates:
- GOTCHAS: replaced the 26B entry with the actual cause; scoped the
original "Thinking Mode Eats Context" entry to single-turn pipelines
- SYNTHESIS: split the "Mandatory Ollama Settings" block into
single-turn vs multi-turn variants; updated anti-patterns and
quick-start checklist
- CORPUS_cli_coding_agent.md: revised pointer and config template
- docs/reference/bakeoff-2026-04-18.md: added Round 3 section with
the correction notice at the top of the file and full diagnostic
methodology
New artifacts: harness_no_think_flag.py, harness_write_no_think.py,
and 4 new log files demonstrating all three models pass when think
is left at default.
Round 2 tested the hypothesis that 26B's silent-stop was about
write_file argument size. Result: refuted.
- Patch-mode (apply_patch instead of write_file): 26B fails identically
at iter 6. Tool-arg size is not the cause.
- Truncation sweep on tool responses reveals the real trigger: cap at
800 or 1200 chars → 26B PASSES (1200-cap is 8.4s, fastest of any run).
Cap at 1600, 2000, or unlimited → 26B silent-stops with eval=4.
Revised understanding: 26B silent-stops when cumulative tool-response
context crosses a shape threshold around 1200-1600 chars per response.
Not a tool-arg bug, not a raw code-gen bug — 26B emits correct code
fine in both one-shot and short-context settings.
Production CLI agents (openclaw, open code, aider) typically truncate
tool responses by default, so this failure may not surface in them.
Custom harnesses should cap ≤1200 chars per tool response when
targeting the 26B MoE.
Updates GOTCHAS (rewritten entry with the truncation sweep table),
SYNTHESIS model-selection row, CORPUS_cli_coding_agent.md pointer,
docs/reference/bakeoff-2026-04-18.md with full Round 2 methodology
and data.
Adds harness_patch.py (apply_patch edit tool), harness_patch_truncated.py
(env-configurable TOOL_RESULT_CAP), all 7 run logs, and a
.secrets.baseline for detect-secrets false positives on JSON timestamps.
Ran minimal agent loop (Ollama /api/chat + read_file/write_file/run_bash) on
steel141 3090 Ti against 3 models on a broken-median-function task:
- gemma4:31b-it-q4_K_M: PASS (8 iters, 1 write, 44s) — textbook trace
- qwen3-coder:30b: PASS (15 iters, 1 write, 22s) — correct but chatty
- gemma4:26b: FAIL (6 iters, 0 writes) — silently stops with eval=4
after reading source. Reproduced on second run. One-shot probe
confirms 26b CAN produce the correct fix — failure is specifically
at the write_file tool-call argument boundary.
Updates GOTCHAS with a new HIGH-severity entry, SYNTHESIS model-selection
table, CORPUS_cli_coding_agent.md empirical-follow-up pointer, and adds
docs/reference/bakeoff-2026-04-18.md with the full writeup.
Fills the gap between existing chat-agent (Simon) and pipeline
(AI_Visualizer) patterns. Covers openclaw/open code/pi/hermes first-party
agents from the HF launch blog, honest positioning vs qwen3-coder:30b,
CLI-agent-specific gotchas (safety filter on security code, long-JSON
weakness, no code exec), and a concrete homelab bakeoff plan pointed at
CT 166 openclaw2 → CT 105 Ollama on pve197.
Key research finding: Google published LiveCodeBench + Codeforces but
NOT SWE-bench or Aider polyglot. The "autonomous agents" claim is
plausible but unproven for multi-file repo-scale coding specifically.
Patches the top-level corpus docs with the 13 findings flagged during the
2026-04-18 canonical tooling research pass. tooling/README.md now marks each
finding [merged: <file>] or [flagged] for provenance.
- CORPUS_ollama_variants.md: annotate gemma4:26b as MoE (25.2B total / 3.8B
active, 8-of-128 experts + 1 shared). Note Q4_K_M inference is standard
(the "MoE quality degrades at 4-bit" caveat is training-only). Add note
that audio on E-series is NOT available via Ollama — llama.cpp mmproj
or vLLM only.
- CORPUS_capabilities.md: native system role, configurable thinking mode,
first trained tool use (vs Gemma 1/2/3 proof-of-concept), native object
detection with bbox output in 1000x1000 coords, pointer to EmbeddingGemma
for retrieval (Gemma 4 has no embedding mode).
- CORPUS_tool_calling_format.md: add Chat Template Context section
documenting the <|turn>/<turn|> asymmetric brackets (new in Gemma 4,
replaced <start_of_turn>/<end_of_turn>) plus <|think>, <|channel>,
<|image>, <|audio> tokens. Add HF transformers Alternative section
showing processor.parse_response with response_schema.
- GOTCHAS.md: add MEDIUM gotcha for abandoned google/gemma_pytorch (no
Gemma 4 support since 2025-05-30). Expand fine-tuning section with FA2/FA4
head_dim=512 break, fused LoRA kernel issues, 26B A4B training-quant
guidance, new tool-call tokens as learned embeddings.
- SYNTHESIS.md: add banner pointing to tooling/ for canonical upstream
material. Add embeddinggemma row to Model Selection table.
Also:
- Add .gitignore excluding .backup/ (local scratch per global CLAUDE.md
convention, not needed in tracked history) and __pycache__/.
- Add .claude/handoffs/2026-04-18-canonical-tooling-research.md so future
sessions can pick up cold — facts verified, open threads, what changed.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Five-lane parallel research pass. Each subdir under tooling/ has its own
README indexing downloaded files with verified upstream sources.
- google-official/: deepmind-gemma JAX examples, gemma_pytorch scripts,
gemma.cpp API server docs, google-gemma/cookbook notebooks, ai.google.dev
HTML snapshots, Gemma 3 tech report
- huggingface/: 8 gemma-4-* model cards, chat-template .jinja files,
tokenizer_config.json, transformers gemma4/ source, launch blog posts,
official HF Spaces app.py
- inference-frameworks/: vLLM/llama.cpp/MLX/Keras-hub/TGI/Gemini API/Vertex AI
comparison, run_commands.sh with 8 working launches, 9 code snippets
- gemma-family/: 12 per-variant briefs (ShieldGemma 2, CodeGemma, PaliGemma 2,
Recurrent/Data/Med/TxGemma, Embedding/Translate/Function/Dolphin/SignGemma)
- fine-tuning/: Unsloth Gemma 4 notebooks, Axolotl YAMLs (incl 26B-A4B MoE),
TRL scripts, Google cookbook fine-tune notebooks, recipe-recommendation.md
Findings that update earlier CORPUS_* docs are flagged in tooling/README.md
(not applied) — notably the new <|turn>/<turn|> prompt format, gemma_pytorch
abandonment, gemma.cpp Gemini-API server, transformers AutoModelForMultimodalLM,
FA2 head_dim=512 break, 26B-A4B MoE quantization rules, no Gemma 4 tech
report PDF yet, no Gemma-4-generation specialized siblings yet.
Pre-commit secrets hook bypassed per user authorization — flagged "secrets"
are base64 notebook cell outputs and example Ed25519 keys in the HDP
agentic-security demo, not real credentials.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Architecture specs, benchmarks, gotchas, Ollama settings, tool calling
format, and implementation patterns from Simon and AI_Visualizer.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Mortdecai