docs: DiffusionGemma research + first-hand smoke test on 3090 Ti
google/diffusiongemma-26B-A4B-it (released 2026-06-10) — Google's first open-weight text-diffusion LLM. Does NOT run in Ollama (unknown arch 'diffusion-gemma'); built llama-diffusion-cli from ggml-org/llama.cpp PR #24423 and smoke-tested Q4_K_M on steel141's 3090 Ti. - New reference doc with specs, build recipe, throughput, and gotchas - CORPUS_ollama_variants.md: "not an Ollama variant" callout - README index line for the reference doc - scripts/diffusiongemma-smoketest/ harness + raw result logs Findings: ~106 tok/s effective / ~2030 tok/s in-step-parallel; correct code + coherent reasoning; <|channel>thought CoT eats the 256-tok canvas so strict short formats need block budgeting. nvidia-smi index != CUDA index on steel141 (select 3090 Ti by UUID). Experimental research artifact, not homelab-deployable until diffusion arch merges to llama.cpp mainline. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Mortdecai
@@ -0,0 +1,128 @@
|
||||
# DiffusionGemma — Research + Smoke Test (2026-06-17)
|
||||
|
||||
> Read when: scoping a text-diffusion model, deciding whether DiffusionGemma is
|
||||
> deployable on the homelab, reproducing the llama-diffusion-cli build, or
|
||||
> comparing diffusion vs autoregressive Gemma 4 throughput.
|
||||
|
||||
## What it is
|
||||
|
||||
`google/diffusiongemma-26B-A4B-it` — released **2026-06-10**, Apache-2.0. Google
|
||||
DeepMind's first open-weight **text-diffusion** LLM. Instead of autoregressive
|
||||
token-by-token decoding, it **denoises a 256-token "canvas" in parallel** over a
|
||||
handful of iterative steps (block-autoregressive multi-canvas sampling).
|
||||
|
||||
| Property | Value |
|
||||
|---|---|
|
||||
| Base | Gemma 4 26B MoE — 25.2B total / **3.8B active** (8 of 128 experts), 30 layers |
|
||||
| Generation | Discrete text diffusion, 256-token canvas, entropy-bound decoder |
|
||||
| Modalities | text + image + video in; **text out** (no audio) |
|
||||
| Context | up to 256K |
|
||||
| Knowledge cutoff | Jan 2025 |
|
||||
| Speed claim | 1,100+ tok/s on H100 FP8; "up to 4×" on constrained consumer HW |
|
||||
| New TF class | `DiffusionGemmaForBlockDiffusion` |
|
||||
| Day-zero serving | vLLM, Transformers, MLX, SGLang |
|
||||
|
||||
## The deployment snag: it does NOT run in your Ollama stack
|
||||
|
||||
Stock `ollama` / `llama-cli` / `llama-server` reject it: **`unknown model
|
||||
architecture: 'diffusion-gemma'`**. Ollama issue #16664 is an unmerged feature
|
||||
request. It needs the dedicated **`llama-diffusion-cli`** binary from
|
||||
**ggml-org/llama.cpp PR #24423** (danielhanchen / Unsloth), which adds the
|
||||
`diffusion-gemma` arch + the entropy-bound canvas decoder. The standard runners
|
||||
cannot generate from it even with that PR — only `llama-diffusion-cli` /
|
||||
`diffusion-gemma-server` can.
|
||||
|
||||
GGUF quants (`unsloth/diffusiongemma-26B-A4B-it-GGUF`): Q4_K_M 16.8 GB · Q5_K_M
|
||||
19.1 · Q6_K 22.7 · Q8_0 26.9 · BF16 50.5.
|
||||
|
||||
## Build recipe (reproduced on steel141, 2026-06-17)
|
||||
|
||||
Prereqs that were **missing** on steel141 and had to be installed:
|
||||
- **CUDA toolkit** (`nvcc`) — none present (Ollama ships runtime libs, not the
|
||||
compiler). Installed `nvidia-cuda-toolkit` 12.4.131 via apt; matches driver
|
||||
CUDA 12.4. ~2.5 GB.
|
||||
- **`gh`** — absent. Used `git fetch origin pull/24423/head:pr-24423` instead.
|
||||
|
||||
```bash
|
||||
# on /mnt/ai_data (ollama-owned; sudo mkdir + chown claude first — 325 GB free there)
|
||||
git clone --depth 1 https://github.com/ggml-org/llama.cpp.git
|
||||
cd llama.cpp
|
||||
git fetch --depth 1 origin pull/24423/head:pr-24423 && git checkout pr-24423
|
||||
cmake -B build -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES="86" -DLLAMA_CURL=OFF -DCMAKE_BUILD_TYPE=Release
|
||||
cmake --build build -j 24 --target llama-diffusion-cli
|
||||
# binary: build/bin/llama-diffusion-cli (sm_86 = 3090 Ti; add 75 for the 1660)
|
||||
|
||||
hf download unsloth/diffusiongemma-26B-A4B-it-GGUF \
|
||||
diffusiongemma-26B-A4B-it-Q4_K_M.gguf --local-dir /mnt/ai_data/diffusiongemma/gguf
|
||||
```
|
||||
|
||||
Local artifacts live at `/mnt/ai_data/diffusiongemma/` (GGUF + llama.cpp build).
|
||||
Harness: `scripts/diffusiongemma-smoketest/run.sh`; raw logs in `results/`.
|
||||
|
||||
### GPU-selection gotcha (cost me two failed runs)
|
||||
|
||||
**nvidia-smi index ≠ CUDA index on steel141.** nvidia-smi lists the 3090 Ti as
|
||||
GPU **1** and the 1660 SUPER as GPU 0; CUDA enumerates them the *other way*
|
||||
(3090 Ti = CUDA device **0**). So `CUDA_VISIBLE_DEVICES=1` selected the 6 GB
|
||||
1660 → `cudaMalloc 16013 MiB ... out of memory`. Fix: **select by UUID** —
|
||||
`CUDA_VISIBLE_DEVICES=GPU-61fed72d-...` (order-independent), or set
|
||||
`CUDA_DEVICE_ORDER=PCI_BUS_ID` and verify. Get the UUID from `nvidia-smi -L`.
|
||||
|
||||
Also: the 16 GB model won't fit alongside a loaded `gemma4:26b` (it splits
|
||||
~14 GB onto the 3090 Ti). For each run I **stopped ollama** (`sudo systemctl
|
||||
stop ollama`), ran, then restarted it — `ollama stop <model>` alone loses the
|
||||
race because live traffic re-loads the model within minutes. Bounded ~3-min
|
||||
windows; ollama recovered clean each time.
|
||||
|
||||
## Smoke-test results (Q4_K_M, 3090 Ti, `--diffusion-eb auto`, `-n 256`)
|
||||
|
||||
| Prompt | Steps/block | ms/step | Effective tok/s | In-step parallel | Verdict |
|
||||
|---|---|---|---|---|---|
|
||||
| reasoning (3 sentences) | 19 | 126 | 106.8 | 2030 tok/s | coherent, on-topic |
|
||||
| code (`is_prime` + docstring) | 13 | 135 | **146.2** | 1900 tok/s | **correct Python**, right algorithm |
|
||||
| format (5 colors, 1 block) | 21 | 124 | 98.6 | 2070 tok/s | truncated mid-thought (see below) |
|
||||
| format (4 blocks, `-n 512`) | 25.5 avg / 2 blocks | 124 | 81.1 | 2068 tok/s | still in thought channel, no final answer |
|
||||
|
||||
- **Model load:** ~7 s (16 GB weights → 3090 Ti). VRAM ~16 GB.
|
||||
- **Diffusion confirmed:** the entropy-bound decoder converges a 256-token canvas
|
||||
in **13–25 steps** (max 48), ~124 ms/step. "In-step parallel" ≈ **1,900–2,070
|
||||
tok/s** is the canvas-parallel rate; **effective 81–146 tok/s** is what you
|
||||
actually get end-to-end. For comparison the AR `gemma4:26b` MoE does ~134 tok/s
|
||||
— DiffusionGemma is in the same ballpark via a totally different mechanism, and
|
||||
would scale better as canvases lengthen.
|
||||
|
||||
### Behavioral finding: the `<|channel>thought` channel eats the canvas
|
||||
|
||||
DiffusionGemma-**it** emits an explicit chain-of-thought block
|
||||
(`<|channel>thought ...`) that is denoised *inside* the canvas. On a single
|
||||
256-token canvas, prompts that trigger long deliberation get **truncated before
|
||||
the final answer**. The "5 primary colors of light" prompt is mildly adversarial
|
||||
(there are only *three*), and the model spiraled — even at 4 blocks / 512 tokens
|
||||
it was still weighing whether five primaries exist and never emitted a list. The
|
||||
reasoning and code prompts, which don't fight the model, came out clean.
|
||||
|
||||
**Takeaway for anyone building on it:** budget canvas blocks for *thought +
|
||||
answer*, not just answer (`--diffusion-blocks` ≥ 4 for non-trivial prompts), or
|
||||
steer/suppress the thought channel. Don't expect tight format adherence in a
|
||||
single canvas.
|
||||
|
||||
## Bottom line
|
||||
|
||||
- DiffusionGemma is **real, runs locally on a single 3090 Ti** (Q4_K_M, ~16 GB)
|
||||
via the PR #24423 `llama-diffusion-cli`, and the diffusion mechanism works.
|
||||
- It is **not deployable through your Ollama stack** today and won't be until the
|
||||
diffusion arch merges into llama.cpp mainline (then Ollama). For now it's an
|
||||
**experimental research artifact**, not a drop-in for Simon / AI_Visualizer /
|
||||
mort-bot.
|
||||
- Generation quality at Q4_K_M is solid on cooperative prompts (correct code,
|
||||
coherent reasoning) but the thought-channel-eats-canvas behavior makes strict
|
||||
short-format outputs unreliable without block budgeting.
|
||||
|
||||
## Sources
|
||||
|
||||
- https://ai.google.dev/gemma/docs/diffusiongemma
|
||||
- https://huggingface.co/google/diffusiongemma-26B-A4B-it
|
||||
- https://huggingface.co/unsloth/diffusiongemma-26B-A4B-it-GGUF
|
||||
- https://github.com/ggml-org/llama.cpp (PR #24423)
|
||||
- https://github.com/ollama/ollama/issues/16664
|
||||
- https://www.theregister.com/ai-and-ml/2026/06/11/googles-diffusiongemma-uses-diffusion-tech-to-speed-text-generation/
|
||||
Reference in New Issue
Block a user