From 2336fb437ead35bc297a121b093f7d712332a6d6 Mon Sep 17 00:00:00 2001 From: Mortdecai Date: Tue, 16 Jun 2026 17:51:30 -0400 Subject: [PATCH] docs: add gemma4:12b variant + smoke-test results - 11.9B dense, 256K ctx, Q4_K_M, vision+audio+tools+thinking - requires Ollama >= 0.30.5; upgraded steel141 0.20.4 -> 0.30.8 to pull - smoke-tested 2026-06-16: text/vision/tools pass at 70 tok/s 100% GPU on 3090 Ti - flag: mort-bakeoff 0.20.4 thinking-in-context pin now needs re-verification - audio capability listed by ollama show but UNVERIFIED through API Co-Authored-By: Claude Opus 4.8 (1M context) --- CORPUS_ollama_variants.md | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/CORPUS_ollama_variants.md b/CORPUS_ollama_variants.md index a1d2d0a..bce90c3 100644 --- a/CORPUS_ollama_variants.md +++ b/CORPUS_ollama_variants.md @@ -1,15 +1,20 @@ # Gemma 4 on Ollama — Available Variants -> Last verified against Seth's homelab: 2026-04-12 +> Last verified against Seth's homelab: 2026-06-16 (12b added; steel141 Ollama upgraded 0.20.4 → 0.30.8) ## Ollama Model Tags | Tag | Params | Quant | Size on Disk | VRAM | Notes | |-----|--------|-------|-------------|------|-------| | `gemma4:e4b-it-q8_0` | ~8B total / 4B effective | Q8_0 | 11.6GB | ~12GB | Vision + audio capable. ~25 tok/s on V100 | +| `gemma4:12b` | 11.9B **dense** | Q4_K_M (default) | 7.6GB | ~8.4GB | **Released 2026-06-03, Apache-2.0.** Encoder-free unified multimodal — `ollama show` lists **vision + audio + tools + thinking**. 256K context (262144). 52.4M CLIP projector. Smoke-tested on steel141 2026-06-16: text reasoning correct, vision accurate (shapes/colors/spatial/OCR), tool call fired on explicit ask — all at **70 tok/s, 100% GPU** on 3090 Ti. Fits a 16GB laptop. **Requires Ollama ≥ 0.30.5** (HTTP 412 on older — this gated the whole 0.20.x homelab). Separate `gemma4:12b-mtp` tag adds Multi-Token-Prediction drafters (1.4–2.2× speculative decode). Google: near-26B-MoE quality at <½ the memory; beats Gemma 3 27B on GPQA Diamond / MMLU-Pro / DocVQA. | | `gemma4:26b` | 25.2B total / **3.8B active (MoE)** | Q4_K_M (default) | 18.0GB | ~18GB | Sweet spot for quality/speed. ~134 tok/s on 3090 Ti. **8 experts active of 128 + 1 shared** — runs at ~4B-speed, hence throughput. Q4_K_M inference is standard (Mixtral/DeepSeek ship same); the "MoE quality degrades at 4-bit" caveat is a **training-time** concern, not inference. See `tooling/huggingface/model-cards/gemma-4-26B-A4B-it-README.md` for the full card. | | `gemma4:31b-it-q4_K_M` | 31.3B | Q4_K_M | 19.9GB | ~24.5GB | Sharpest but 5x slower (~28 tok/s on 3090 Ti, memory pressure) | +> **Throughput note:** `gemma4:12b` (70 tok/s) is *slower* per-token than `gemma4:26b` (~134 tok/s) despite being smaller — because 26b is MoE (only ~3.8B params active per token) while 12b is fully dense (all 11.9B active). Pick 12b for quality-per-VRAM and audio on a small box; pick 26b for raw speed when you have the VRAM. + +> **⚠️ Ollama 0.20.4 pin invalidated (2026-06-16):** steel141 was upgraded 0.20.4 → 0.30.8 to pull `gemma4:12b`. The `mort-bakeoff-2026-04-18.md` finding ("thinking does NOT accumulate in context **on Ollama 0.20.4**") was pinned to that version and must be **re-verified on 0.30.8** before relying on it. Binary backup + override saved in `.backup/ollama-bin-0.20.4-*`. + ## Capabilities by Variant (from `ollama show`) All variants support: @@ -21,6 +26,8 @@ All variants support: E-series (E2B, E4B) additionally support: - Audio input (conformer encoder) — **but not via Ollama**; requires llama.cpp with the `mmproj-*-E*B-it-*.gguf` projector, or vLLM's `input_features_padded`. See `tooling/inference-frameworks/README.md`. +`gemma4:12b` `ollama show` lists **audio** as a capability (encoder-free design projects raw waveforms directly). **UNVERIFIED through the Ollama API** — the 2026-06-16 smoke test covered text, vision, and tools only. Before relying on 12b audio, confirm Ollama 0.30.8 actually accepts an audio payload (historically it rejected audio for the E-series even when `ollama show` advertised it). + ## GPU Coexistence (pve197 V100 32GB) - gemma4:26b + SDXL Turbo: ~28.5GB peak VRAM — fits on V100-32GB