# DiffusionGemma — Research + Smoke Test (2026-06-17)

> Read when: scoping a text-diffusion model, deciding whether DiffusionGemma is
> deployable on the homelab, reproducing the llama-diffusion-cli build, or
> comparing diffusion vs autoregressive Gemma 4 throughput.

## What it is

`google/diffusiongemma-26B-A4B-it` — released **2026-06-10**, Apache-2.0. Google
DeepMind's first open-weight **text-diffusion** LLM. Instead of autoregressive
token-by-token decoding, it **denoises a 256-token "canvas" in parallel** over a
handful of iterative steps (block-autoregressive multi-canvas sampling).

| Property | Value |
|---|---|
| Base | Gemma 4 26B MoE — 25.2B total / **3.8B active** (8 of 128 experts), 30 layers |
| Generation | Discrete text diffusion, 256-token canvas, entropy-bound decoder |
| Modalities | text + image + video in; **text out** (no audio) |
| Context | up to 256K | 
| Knowledge cutoff | Jan 2025 |
| Speed claim | 1,100+ tok/s on H100 FP8; "up to 4×" on constrained consumer HW |
| New TF class | `DiffusionGemmaForBlockDiffusion` |
| Day-zero serving | vLLM, Transformers, MLX, SGLang |

## The deployment snag: it does NOT run in your Ollama stack

Stock `ollama` / `llama-cli` / `llama-server` reject it: **`unknown model
architecture: 'diffusion-gemma'`**. Ollama issue #16664 is an unmerged feature
request. It needs the dedicated **`llama-diffusion-cli`** binary from
**ggml-org/llama.cpp PR #24423** (danielhanchen / Unsloth), which adds the
`diffusion-gemma` arch + the entropy-bound canvas decoder. The standard runners
cannot generate from it even with that PR — only `llama-diffusion-cli` /
`diffusion-gemma-server` can.

GGUF quants (`unsloth/diffusiongemma-26B-A4B-it-GGUF`): Q4_K_M 16.8 GB · Q5_K_M
19.1 · Q6_K 22.7 · Q8_0 26.9 · BF16 50.5.

## Build recipe (reproduced on steel141, 2026-06-17)

Prereqs that were **missing** on steel141 and had to be installed:
- **CUDA toolkit** (`nvcc`) — none present (Ollama ships runtime libs, not the
  compiler). Installed `nvidia-cuda-toolkit` 12.4.131 via apt; matches driver
  CUDA 12.4. ~2.5 GB.
- **`gh`** — absent. Used `git fetch origin pull/24423/head:pr-24423` instead.

```bash
# on /mnt/ai_data (ollama-owned; sudo mkdir + chown claude first — 325 GB free there)
git clone --depth 1 https://github.com/ggml-org/llama.cpp.git
cd llama.cpp
git fetch --depth 1 origin pull/24423/head:pr-24423 && git checkout pr-24423
cmake -B build -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES="86" -DLLAMA_CURL=OFF -DCMAKE_BUILD_TYPE=Release
cmake --build build -j 24 --target llama-diffusion-cli
# binary: build/bin/llama-diffusion-cli   (sm_86 = 3090 Ti; add 75 for the 1660)

hf download unsloth/diffusiongemma-26B-A4B-it-GGUF \
  diffusiongemma-26B-A4B-it-Q4_K_M.gguf --local-dir /mnt/ai_data/diffusiongemma/gguf
```

Local artifacts live at `/mnt/ai_data/diffusiongemma/` (GGUF + llama.cpp build).
Harness: `scripts/diffusiongemma-smoketest/run.sh`; raw logs in `results/`.

### GPU-selection gotcha (cost me two failed runs)

**nvidia-smi index ≠ CUDA index on steel141.** nvidia-smi lists the 3090 Ti as
GPU **1** and the 1660 SUPER as GPU 0; CUDA enumerates them the *other way*
(3090 Ti = CUDA device **0**). So `CUDA_VISIBLE_DEVICES=1` selected the 6 GB
1660 → `cudaMalloc 16013 MiB ... out of memory`. Fix: **select by UUID** —
`CUDA_VISIBLE_DEVICES=GPU-61fed72d-...` (order-independent), or set
`CUDA_DEVICE_ORDER=PCI_BUS_ID` and verify. Get the UUID from `nvidia-smi -L`.

Also: the 16 GB model won't fit alongside a loaded `gemma4:26b` (it splits
~14 GB onto the 3090 Ti). For each run I **stopped ollama** (`sudo systemctl
stop ollama`), ran, then restarted it — `ollama stop <model>` alone loses the
race because live traffic re-loads the model within minutes. Bounded ~3-min
windows; ollama recovered clean each time.

## Smoke-test results (Q4_K_M, 3090 Ti, `--diffusion-eb auto`, `-n 256`)

| Prompt | Steps/block | ms/step | Effective tok/s | In-step parallel | Verdict |
|---|---|---|---|---|---|
| reasoning (3 sentences) | 19 | 126 | 106.8 | 2030 tok/s | coherent, on-topic |
| code (`is_prime` + docstring) | 13 | 135 | **146.2** | 1900 tok/s | **correct Python**, right algorithm |
| format (5 colors, 1 block) | 21 | 124 | 98.6 | 2070 tok/s | truncated mid-thought (see below) |
| format (4 blocks, `-n 512`) | 25.5 avg / 2 blocks | 124 | 81.1 | 2068 tok/s | still in thought channel, no final answer |

- **Model load:** ~7 s (16 GB weights → 3090 Ti). VRAM ~16 GB.
- **Diffusion confirmed:** the entropy-bound decoder converges a 256-token canvas
  in **13–25 steps** (max 48), ~124 ms/step. "In-step parallel" ≈ **1,900–2,070
  tok/s** is the canvas-parallel rate; **effective 81–146 tok/s** is what you
  actually get end-to-end. For comparison the AR `gemma4:26b` MoE does ~134 tok/s
  — DiffusionGemma is in the same ballpark via a totally different mechanism, and
  would scale better as canvases lengthen.

### Behavioral finding: the `<|channel>thought` channel eats the canvas

DiffusionGemma-**it** emits an explicit chain-of-thought block
(`<|channel>thought ...`) that is denoised *inside* the canvas. On a single
256-token canvas, prompts that trigger long deliberation get **truncated before
the final answer**. The "5 primary colors of light" prompt is mildly adversarial
(there are only *three*), and the model spiraled — even at 4 blocks / 512 tokens
it was still weighing whether five primaries exist and never emitted a list. The
reasoning and code prompts, which don't fight the model, came out clean.

**Takeaway for anyone building on it:** budget canvas blocks for *thought +
answer*, not just answer (`--diffusion-blocks` ≥ 4 for non-trivial prompts), or
steer/suppress the thought channel. Don't expect tight format adherence in a
single canvas.

## Bottom line

- DiffusionGemma is **real, runs locally on a single 3090 Ti** (Q4_K_M, ~16 GB)
  via the PR #24423 `llama-diffusion-cli`, and the diffusion mechanism works.
- It is **not deployable through your Ollama stack** today and won't be until the
  diffusion arch merges into llama.cpp mainline (then Ollama). For now it's an
  **experimental research artifact**, not a drop-in for Simon / AI_Visualizer /
  mort-bot.
- Generation quality at Q4_K_M is solid on cooperative prompts (correct code,
  coherent reasoning) but the thought-channel-eats-canvas behavior makes strict
  short-format outputs unreliable without block budgeting.

## Sources

- https://ai.google.dev/gemma/docs/diffusiongemma
- https://huggingface.co/google/diffusiongemma-26B-A4B-it
- https://huggingface.co/unsloth/diffusiongemma-26B-A4B-it-GGUF
- https://github.com/ggml-org/llama.cpp (PR #24423)
- https://github.com/ollama/ollama/issues/16664
- https://www.theregister.com/ai-and-ml/2026/06/11/googles-diffusiongemma-uses-diffusion-tech-to-speed-text-generation/