gemma4-research/docs/reference/diffusiongemma-smoketest-2026-06-17.md

DiffusionGemma — Research + Smoke Test (2026-06-17)

Read when: scoping a text-diffusion model, deciding whether DiffusionGemma is deployable on the homelab, reproducing the llama-diffusion-cli build, or comparing diffusion vs autoregressive Gemma 4 throughput.

What it is

google/diffusiongemma-26B-A4B-it — released 2026-06-10, Apache-2.0. Google DeepMind's first open-weight text-diffusion LLM. Instead of autoregressive token-by-token decoding, it denoises a 256-token "canvas" in parallel over a handful of iterative steps (block-autoregressive multi-canvas sampling).

Property	Value
Base	Gemma 4 26B MoE — 25.2B total / 3.8B active (8 of 128 experts), 30 layers
Generation	Discrete text diffusion, 256-token canvas, entropy-bound decoder
Modalities	text + image + video in; text out (no audio)
Context	up to 256K
Knowledge cutoff	Jan 2025
Speed claim	1,100+ tok/s on H100 FP8; "up to 4×" on constrained consumer HW
New TF class	DiffusionGemmaForBlockDiffusion
Day-zero serving	vLLM, Transformers, MLX, SGLang

The deployment snag: it does NOT run in your Ollama stack

Stock ollama / llama-cli / llama-server reject it: unknown model architecture: 'diffusion-gemma'. Ollama issue #16664 is an unmerged feature request. It needs the dedicated llama-diffusion-cli binary from ggml-org/llama.cpp PR #24423 (danielhanchen / Unsloth), which adds the diffusion-gemma arch + the entropy-bound canvas decoder. The standard runners cannot generate from it even with that PR — only llama-diffusion-cli / diffusion-gemma-server can.

GGUF quants (unsloth/diffusiongemma-26B-A4B-it-GGUF): Q4_K_M 16.8 GB · Q5_K_M 19.1 · Q6_K 22.7 · Q8_0 26.9 · BF16 50.5.

Build recipe (reproduced on steel141, 2026-06-17)

Prereqs that were missing on steel141 and had to be installed:

• CUDA toolkit (nvcc) — none present (Ollama ships runtime libs, not the compiler). Installed nvidia-cuda-toolkit 12.4.131 via apt; matches driver CUDA 12.4. ~2.5 GB.
• gh — absent. Used git fetch origin pull/24423/head:pr-24423 instead.

# on /mnt/ai_data (ollama-owned; sudo mkdir + chown claude first — 325 GB free there)
git clone --depth 1 https://github.com/ggml-org/llama.cpp.git
cd llama.cpp
git fetch --depth 1 origin pull/24423/head:pr-24423 && git checkout pr-24423
cmake -B build -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES="86" -DLLAMA_CURL=OFF -DCMAKE_BUILD_TYPE=Release
cmake --build build -j 24 --target llama-diffusion-cli
# binary: build/bin/llama-diffusion-cli   (sm_86 = 3090 Ti; add 75 for the 1660)

hf download unsloth/diffusiongemma-26B-A4B-it-GGUF \
  diffusiongemma-26B-A4B-it-Q4_K_M.gguf --local-dir /mnt/ai_data/diffusiongemma/gguf

Local artifacts live at /mnt/ai_data/diffusiongemma/ (GGUF + llama.cpp build). Harness: scripts/diffusiongemma-smoketest/run.sh; raw logs in results/.

GPU-selection gotcha (cost me two failed runs)

nvidia-smi index ≠ CUDA index on steel141. nvidia-smi lists the 3090 Ti as GPU 1 and the 1660 SUPER as GPU 0; CUDA enumerates them the other way (3090 Ti = CUDA device 0). So CUDA_VISIBLE_DEVICES=1 selected the 6 GB 1660 → cudaMalloc 16013 MiB ... out of memory. Fix: select by UUID — CUDA_VISIBLE_DEVICES=GPU-61fed72d-... (order-independent), or set CUDA_DEVICE_ORDER=PCI_BUS_ID and verify. Get the UUID from nvidia-smi -L.

Also: the 16 GB model won't fit alongside a loaded gemma4:26b (it splits ~14 GB onto the 3090 Ti). For each run I stopped ollama (sudo systemctl stop ollama), ran, then restarted it — ollama stop <model> alone loses the race because live traffic re-loads the model within minutes. Bounded ~3-min windows; ollama recovered clean each time.

Smoke-test results (Q4_K_M, 3090 Ti, --diffusion-eb auto, -n 256)

Prompt	Steps/block	ms/step	Effective tok/s	In-step parallel	Verdict
reasoning (3 sentences)	19	126	106.8	2030 tok/s	coherent, on-topic
code (is_prime + docstring)	13	135	146.2	1900 tok/s	correct Python, right algorithm
format (5 colors, 1 block)	21	124	98.6	2070 tok/s	truncated mid-thought (see below)
format (4 blocks, -n 512)	25.5 avg / 2 blocks	124	81.1	2068 tok/s	still in thought channel, no final answer

• Model load: ~7 s (16 GB weights → 3090 Ti). VRAM ~16 GB.
• Diffusion confirmed: the entropy-bound decoder converges a 256-token canvas in 13–25 steps (max 48), ~124 ms/step. "In-step parallel" ≈ 1,900–2,070 tok/s is the canvas-parallel rate; effective 81–146 tok/s is what you actually get end-to-end. For comparison the AR gemma4:26b MoE does ~134 tok/s — DiffusionGemma is in the same ballpark via a totally different mechanism, and would scale better as canvases lengthen.

Behavioral finding: the <|channel>thought channel eats the canvas

DiffusionGemma-it emits an explicit chain-of-thought block (<|channel>thought ...) that is denoised inside the canvas. On a single 256-token canvas, prompts that trigger long deliberation get truncated before the final answer. The "5 primary colors of light" prompt is mildly adversarial (there are only three), and the model spiraled — even at 4 blocks / 512 tokens it was still weighing whether five primaries exist and never emitted a list. The reasoning and code prompts, which don't fight the model, came out clean.

Takeaway for anyone building on it: budget canvas blocks for thought + answer, not just answer (--diffusion-blocks ≥ 4 for non-trivial prompts), or steer/suppress the thought channel. Don't expect tight format adherence in a single canvas.

Bottom line

• DiffusionGemma is real, runs locally on a single 3090 Ti (Q4_K_M, ~16 GB) via the PR #24423 llama-diffusion-cli, and the diffusion mechanism works.
• It is not deployable through your Ollama stack today and won't be until the diffusion arch merges into llama.cpp mainline (then Ollama). For now it's an experimental research artifact, not a drop-in for Simon / AI_Visualizer / mort-bot.
• Generation quality at Q4_K_M is solid on cooperative prompts (correct code, coherent reasoning) but the thought-channel-eats-canvas behavior makes strict short-format outputs unreliable without block budgeting.

Sources

• https://ai.google.dev/gemma/docs/diffusiongemma
• https://huggingface.co/google/diffusiongemma-26B-A4B-it
• https://huggingface.co/unsloth/diffusiongemma-26B-A4B-it-GGUF
• https://github.com/ggml-org/llama.cpp (PR #24423)
• https://github.com/ollama/ollama/issues/16664
• https://www.theregister.com/ai-and-ml/2026/06/11/googles-diffusiongemma-uses-diffusion-tech-to-speed-text-generation/