Mortdecai
6bce2d4c3b
google/diffusiongemma-26B-A4B-it (released 2026-06-10) — Google's first open-weight text-diffusion LLM. Does NOT run in Ollama (unknown arch 'diffusion-gemma'); built llama-diffusion-cli from ggml-org/llama.cpp PR #24423 and smoke-tested Q4_K_M on steel141's 3090 Ti. - New reference doc with specs, build recipe, throughput, and gotchas - CORPUS_ollama_variants.md: "not an Ollama variant" callout - README index line for the reference doc - scripts/diffusiongemma-smoketest/ harness + raw result logs Findings: ~106 tok/s effective / ~2030 tok/s in-step-parallel; correct code + coherent reasoning; <|channel>thought CoT eats the 256-tok canvas so strict short formats need block budgeting. nvidia-smi index != CUDA index on steel141 (select 3090 Ti by UUID). Experimental research artifact, not homelab-deployable until diffusion arch merges to llama.cpp mainline. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
53 lines
5.1 KiB
Markdown
53 lines
5.1 KiB
Markdown
# Gemma 4 on Ollama — Available Variants
|
||
|
||
> Last verified against Seth's homelab: 2026-06-16 (12b added; steel141 Ollama upgraded 0.20.4 → 0.30.8)
|
||
|
||
## Ollama Model Tags
|
||
|
||
| Tag | Params | Quant | Size on Disk | VRAM | Notes |
|
||
|-----|--------|-------|-------------|------|-------|
|
||
| `gemma4:e4b-it-q8_0` | ~8B total / 4B effective | Q8_0 | 11.6GB | ~12GB | Vision + audio capable. ~25 tok/s on V100 |
|
||
| `gemma4:12b` | 11.9B **dense** | Q4_K_M (default) | 7.6GB | ~8.4GB | **Released 2026-06-03, Apache-2.0.** Encoder-free unified multimodal — `ollama show` lists **vision + audio + tools + thinking**. 256K context (262144). 52.4M CLIP projector. Smoke-tested on steel141 2026-06-16: text reasoning correct, vision accurate (shapes/colors/spatial/OCR), tool call fired on explicit ask — all at **70 tok/s, 100% GPU** on 3090 Ti. Fits a 16GB laptop. **Requires Ollama ≥ 0.30.5** (HTTP 412 on older — this gated the whole 0.20.x homelab). Separate `gemma4:12b-mtp` tag adds Multi-Token-Prediction drafters (1.4–2.2× speculative decode). Google: near-26B-MoE quality at <½ the memory; beats Gemma 3 27B on GPQA Diamond / MMLU-Pro / DocVQA. |
|
||
| `gemma4:26b` | 25.2B total / **3.8B active (MoE)** | Q4_K_M (default) | 18.0GB | ~18GB | Sweet spot for quality/speed. ~134 tok/s on 3090 Ti. **8 experts active of 128 + 1 shared** — runs at ~4B-speed, hence throughput. Q4_K_M inference is standard (Mixtral/DeepSeek ship same); the "MoE quality degrades at 4-bit" caveat is a **training-time** concern, not inference. See `tooling/huggingface/model-cards/gemma-4-26B-A4B-it-README.md` for the full card. |
|
||
| `gemma4:31b-it-q4_K_M` | 31.3B | Q4_K_M | 19.9GB | ~24.5GB | Sharpest but 5x slower (~28 tok/s on 3090 Ti, memory pressure) |
|
||
|
||
> **Throughput note:** `gemma4:12b` (70 tok/s) is *slower* per-token than `gemma4:26b` (~134 tok/s) despite being smaller — because 26b is MoE (only ~3.8B params active per token) while 12b is fully dense (all 11.9B active). Pick 12b for quality-per-VRAM and audio on a small box; pick 26b for raw speed when you have the VRAM.
|
||
|
||
> **⚠️ Ollama 0.20.4 pin invalidated (2026-06-16):** steel141 was upgraded 0.20.4 → 0.30.8 to pull `gemma4:12b`. The `mort-bakeoff-2026-04-18.md` finding ("thinking does NOT accumulate in context **on Ollama 0.20.4**") was pinned to that version and must be **re-verified on 0.30.8** before relying on it. Binary backup + override saved in `.backup/ollama-bin-0.20.4-*`.
|
||
|
||
> **DiffusionGemma is NOT an Ollama variant (do not try to pull it).** `google/diffusiongemma-26B-A4B-it` (released 2026-06-10, the 26B-A4B MoE backbone + a text-**diffusion** head) is a separate generation paradigm — it denoises a 256-token canvas in parallel rather than decoding token-by-token. Ollama/`llama-cli`/`llama-server` reject it (`unknown model architecture: 'diffusion-gemma'`); it needs the dedicated `llama-diffusion-cli` from **ggml-org/llama.cpp PR #24423**. Smoke-tested locally on steel141's 3090 Ti 2026-06-17 (Q4_K_M, ~16 GB, ~106 tok/s effective / ~2030 tok/s in-step-parallel; correct code, coherent reasoning; the `<|channel>thought` CoT eats the canvas so strict short formats need block budgeting). Experimental research artifact, **not deployable through the homelab Ollama stack** until the diffusion arch merges to llama.cpp mainline. Full writeup + build recipe: `docs/reference/diffusiongemma-smoketest-2026-06-17.md`.
|
||
|
||
## Capabilities by Variant (from `ollama show`)
|
||
|
||
All variants support:
|
||
- Text generation (completion, chat)
|
||
- Vision (image input via base64 in `images` field)
|
||
- Tool/function calling (native Ollama tool format)
|
||
- Thinking (configurable — `ollama show` lists it; Seth's finding is to leave it `false` for tool-use workloads)
|
||
|
||
E-series (E2B, E4B) additionally support:
|
||
- Audio input (conformer encoder) — **but not via Ollama**; requires llama.cpp with the `mmproj-*-E*B-it-*.gguf` projector, or vLLM's `input_features_padded`. See `tooling/inference-frameworks/README.md`.
|
||
|
||
`gemma4:12b` `ollama show` lists **audio** as a capability (encoder-free design projects raw waveforms directly). **UNVERIFIED through the Ollama API** — the 2026-06-16 smoke test covered text, vision, and tools only. Before relying on 12b audio, confirm Ollama 0.30.8 actually accepts an audio payload (historically it rejected audio for the E-series even when `ollama show` advertised it).
|
||
|
||
## GPU Coexistence (pve197 V100 32GB)
|
||
|
||
- gemma4:26b + SDXL Turbo: ~28.5GB peak VRAM — fits on V100-32GB
|
||
- gemma4:31b: 24.5GB alone — memory pressure with any coexisting model
|
||
- gemma4:e4b-it-q8_0: ~12GB — comfortable headroom
|
||
|
||
## Ollama API Endpoint
|
||
|
||
- `/api/generate` (single-turn, used by AI_Visualizer)
|
||
- `/api/chat` (multi-turn with message history, used by Simon)
|
||
- Both accept `tools`, `images`, `stream`, `options`, `keep_alive`
|
||
|
||
## Important Ollama Defaults to Override
|
||
|
||
| Parameter | Ollama Default | Recommended | Why |
|
||
|-----------|---------------|-------------|-----|
|
||
| `num_ctx` | 2048 | 4096-32768 | Default is absurdly small, causes truncation |
|
||
| `num_predict` | 128 | 512-4096+ | Default truncates almost all useful output |
|
||
| `think` | true (Ollama 0.20+) | false | See GOTCHAS doc |
|
||
| `keep_alive` | 5m | 30m-4h | Prevents expensive model reload between calls |
|