Mortdecai
b6190357ba
Cross-host Gemma 4 throughput comparison across three architectures. Harness at scripts/gpu-bakeoff/; writeup at docs/reference/gpu-bakeoff-2026-04-20.md. Key findings: - RTX 3090 Ti wins decode decisively (128 tok/s on gemma4:26b MoE Q4, ~4.7× faster than gemma4:31b dense on the same card). - AMD Strix Halo iGPU lands at ~42% of 3090 Ti decode on ~25% of the memory bandwidth — good SIMD utilization, especially for MoE. - V100 numbers are DEGRADED: CT 167 ai-visualizer SDXL consumes 31/32 GB of its VRAM, forcing Gemma 4 models 95% onto CPU. Isolated V100 run requires SDXL eviction — left as follow-up. - MoE vs dense is the dominant latency factor across all GPUs: ~4 B active params of gemma4:26b beats 31.3 B active of gemma4:31b by the same ratio (~4.7×) on every card tested. Methodology: 1 warmup + 3 measurement runs per (host × model × prompt-length), Ollama's canonical timing fields, temp=0 greedy, num_predict=256. All three Ollama servers accessed via HTTP (Strix via Tailscale). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
32 lines
4.3 KiB
Markdown
32 lines
4.3 KiB
Markdown
# gemma4-research
|
||
|
||
Research corpus and implementation guidance for Google Gemma 4, based on production use in Seth's homelab.
|
||
|
||
## Files
|
||
|
||
| File | What | When to Read |
|
||
|------|------|-------------|
|
||
| `SYNTHESIS.md` | **Start here.** Opinionated guide — how to build with Gemma 4 | Before any new Gemma 4 implementation |
|
||
| `GOTCHAS.md` | Known issues and workarounds, severity-ranked | When debugging Gemma 4 issues or starting a new project |
|
||
| `IMPLEMENTATIONS.md` | Patterns from Simon and AI_Visualizer | When designing a new Gemma 4 integration |
|
||
| `CORPUS_architecture.md` | Model architecture details (layers, attention, PLE, MoE) | When you need to understand WHY Gemma 4 behaves a certain way |
|
||
| `CORPUS_ollama_variants.md` | Available models, sizes, VRAM, Ollama settings | When choosing a model variant or configuring Ollama |
|
||
| `CORPUS_capabilities.md` | Modalities (vision, audio, video, tools), what it can/can't do | When scoping what Gemma 4 can handle |
|
||
| `CORPUS_benchmarks.md` | Full benchmark table vs Gemma 3, arena scores, agentic scores | When comparing Gemma 4 to alternatives |
|
||
| `CORPUS_tool_calling_format.md` | Native token format + JSON API format for function calling | When implementing tool calling |
|
||
| `CORPUS_cli_coding_agent.md` | Positioning Gemma 4 for CLI coding agent use (openclaw / open code / pi / hermes / aider style). Honest take on what Google did and didn't measure, head-to-head with `qwen3-coder:30b`, homelab setup pointer | When scoping a CLI coding agent or deciding Gemma 4 vs Qwen3-Coder |
|
||
| `docs/openwebui-setup.md` | How to configure Gemma 4 inside OpenWebUI — per-setting reference, two ready-to-bake Workspace Model profiles (chat + extract), and a symptom→cause troubleshooting table mapped back to GOTCHAS.md. Assumes Ollama + OpenWebUI are already running. | When setting up or debugging a Gemma 4 model in OpenWebUI, or handing the front-end config to someone else |
|
||
| `docs/reference/bakeoff-2026-04-18.md` | CLI-coding-agent bakeoff on 3090 Ti. **Rounds 1/2 misidentified the cause; Round 3 (the correct one): `think: false` silent-stops gemma4:26b at certain multi-turn states on 32K context.** 31B and Qwen3-Coder robust to the flag. Harness at `scripts/bakeoff/` | When deciding which model to back a CLI agent with, writing a custom agent payload, or debugging a silent tool-call halt |
|
||
| `docs/reference/mort-bakeoff-2026-04-18.md` | mort-bot-specific `think=true` vs `think=false` bakeoff on mort's actual loop shape (gemma4:26b, num_ctx=8192). **Thinking does NOT accumulate in context on Ollama 0.20.4** — strips it from serialized history. Both settings behave identically on step counts, tool counts, wall clock. Harness at `scripts/mort-bakeoff/` | When deciding mort-bot's THINK env var, or when someone claims "think=true eats context" without pinning an Ollama version |
|
||
| `docs/reference/gpu-bakeoff-2026-04-20.md` | Cross-GPU throughput bakeoff: steel141 RTX 3090 Ti vs pve197 V100 vs matt-strix (AMD Strix Halo). **3090 Ti wins decode decisively (128 tok/s on 26B MoE). Strix gets ~42% of that on ~25% of the bandwidth. V100 numbers are degraded because SDXL on CT 167 occupies 31/32 GB of its VRAM.** Also quantifies the MoE vs dense gap: 26B decodes ~4.7× faster than 31B on every card. Harness at `scripts/gpu-bakeoff/` | When choosing which host to run a Gemma 4 workload on, or deciding whether the V100 needs isolated for a given job |
|
||
| `tooling/` | **Canonical upstream tooling** — real scripts, notebooks, model cards, and configs pulled from Google / HF / framework maintainers (147 files). Subdirs: `google-official/`, `huggingface/`, `inference-frameworks/`, `gemma-family/`, `fine-tuning/`. See `tooling/README.md` for index and findings that update the older `CORPUS_*` docs | When you need authoritative source material — model cards, chat templates, fine-tuning recipes, serving commands for vLLM / llama.cpp / MLX, or to scope a specialized sibling (ShieldGemma, EmbeddingGemma, etc.) |
|
||
|
||
## Source Projects
|
||
|
||
- **Simon** (`~/bin/FreibergFamily/simon/`) — Multi-turn chat agent with 6 tools, genealogy historian
|
||
- **AI Visualizer** (`~/bin/AI_Visualizer/`) — Music video generator, 4-stage Gemma pipeline + vision
|
||
|
||
## Key Insight
|
||
|
||
Gemma 4 is ultra-compliant and highly capable but doesn't know who it is. It needs explicit system prompts, not hand-holding. Due to fast local inference, sequential tool calls beat long JSON requests.
|