docs: DiffusionGemma research + first-hand smoke test on 3090 Ti
google/diffusiongemma-26B-A4B-it (released 2026-06-10) — Google's first open-weight text-diffusion LLM. Does NOT run in Ollama (unknown arch 'diffusion-gemma'); built llama-diffusion-cli from ggml-org/llama.cpp PR #24423 and smoke-tested Q4_K_M on steel141's 3090 Ti. - New reference doc with specs, build recipe, throughput, and gotchas - CORPUS_ollama_variants.md: "not an Ollama variant" callout - README index line for the reference doc - scripts/diffusiongemma-smoketest/ harness + raw result logs Findings: ~106 tok/s effective / ~2030 tok/s in-step-parallel; correct code + coherent reasoning; <|channel>thought CoT eats the 256-tok canvas so strict short formats need block budgeting. nvidia-smi index != CUDA index on steel141 (select 3090 Ti by UUID). Experimental research artifact, not homelab-deployable until diffusion arch merges to llama.cpp mainline. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Mortdecai
@@ -15,6 +15,8 @@
|
||||
|
||||
> **⚠️ Ollama 0.20.4 pin invalidated (2026-06-16):** steel141 was upgraded 0.20.4 → 0.30.8 to pull `gemma4:12b`. The `mort-bakeoff-2026-04-18.md` finding ("thinking does NOT accumulate in context **on Ollama 0.20.4**") was pinned to that version and must be **re-verified on 0.30.8** before relying on it. Binary backup + override saved in `.backup/ollama-bin-0.20.4-*`.
|
||||
|
||||
> **DiffusionGemma is NOT an Ollama variant (do not try to pull it).** `google/diffusiongemma-26B-A4B-it` (released 2026-06-10, the 26B-A4B MoE backbone + a text-**diffusion** head) is a separate generation paradigm — it denoises a 256-token canvas in parallel rather than decoding token-by-token. Ollama/`llama-cli`/`llama-server` reject it (`unknown model architecture: 'diffusion-gemma'`); it needs the dedicated `llama-diffusion-cli` from **ggml-org/llama.cpp PR #24423**. Smoke-tested locally on steel141's 3090 Ti 2026-06-17 (Q4_K_M, ~16 GB, ~106 tok/s effective / ~2030 tok/s in-step-parallel; correct code, coherent reasoning; the `<|channel>thought` CoT eats the canvas so strict short formats need block budgeting). Experimental research artifact, **not deployable through the homelab Ollama stack** until the diffusion arch merges to llama.cpp mainline. Full writeup + build recipe: `docs/reference/diffusiongemma-smoketest-2026-06-17.md`.
|
||||
|
||||
## Capabilities by Variant (from `ollama show`)
|
||||
|
||||
All variants support:
|
||||
|
||||
@@ -20,6 +20,7 @@ Research corpus and implementation guidance for Google Gemma 4, based on product
|
||||
| `docs/openwebui-setup.md` | How to configure Gemma 4 inside OpenWebUI — per-setting reference, two ready-to-bake Workspace Model profiles (chat + extract), and a symptom→cause troubleshooting table mapped back to GOTCHAS.md. Assumes Ollama + OpenWebUI are already running. | When setting up or debugging a Gemma 4 model in OpenWebUI, or handing the front-end config to someone else |
|
||||
| `docs/reference/bakeoff-2026-04-18.md` | CLI-coding-agent bakeoff on 3090 Ti. **Rounds 1/2 misidentified the cause; Round 3 (the correct one): `think: false` silent-stops gemma4:26b at certain multi-turn states on 32K context.** 31B and Qwen3-Coder robust to the flag. Harness at `scripts/bakeoff/` | When deciding which model to back a CLI agent with, writing a custom agent payload, or debugging a silent tool-call halt |
|
||||
| `docs/reference/mort-bakeoff-2026-04-18.md` | mort-bot-specific `think=true` vs `think=false` bakeoff on mort's actual loop shape (gemma4:26b, num_ctx=8192). **Thinking does NOT accumulate in context on Ollama 0.20.4** — strips it from serialized history. Both settings behave identically on step counts, tool counts, wall clock. Harness at `scripts/mort-bakeoff/` | When deciding mort-bot's THINK env var, or when someone claims "think=true eats context" without pinning an Ollama version |
|
||||
| `docs/reference/diffusiongemma-smoketest-2026-06-17.md` | **DiffusionGemma** (`google/diffusiongemma-26B-A4B-it`, released 2026-06-10) — Google's first open-weight text-**diffusion** LLM (26B-A4B MoE + canvas diffusion head). Research + first-hand smoke test on steel141's 3090 Ti: does NOT run in Ollama (`unknown model architecture: 'diffusion-gemma'`), needs `llama-diffusion-cli` from ggml-org/llama.cpp PR #24423. Build recipe, throughput (~106 tok/s effective / ~2030 tok/s in-step-parallel at Q4_K_M), the nvidia-smi-vs-CUDA device-ordering gotcha, and the "thought channel eats the canvas" behavior. | When evaluating a text-diffusion model, deciding if DiffusionGemma is homelab-deployable, or reproducing the llama-diffusion-cli build |
|
||||
| `docs/reference/gpu-bakeoff-2026-04-20.md` | Cross-GPU throughput bakeoff: steel141 RTX 3090 Ti vs strix-halo (AMD Strix Halo). **3090 Ti wins decode decisively (128 tok/s on 26B MoE). Strix gets ~42% of that on ~25% of the bandwidth.** Also quantifies the MoE vs dense gap: 26B decodes ~4.7× faster than 31B on both cards. Harness at `scripts/gpu-bakeoff/` | When choosing which host to run a Gemma 4 workload on |
|
||||
| `tooling/` | **Canonical upstream tooling** — real scripts, notebooks, model cards, and configs pulled from Google / HF / framework maintainers (147 files). Subdirs: `google-official/`, `huggingface/`, `inference-frameworks/`, `gemma-family/`, `fine-tuning/`. See `tooling/README.md` for index and findings that update the older `CORPUS_*` docs | When you need authoritative source material — model cards, chat templates, fine-tuning recipes, serving commands for vLLM / llama.cpp / MLX, or to scope a specialized sibling (ShieldGemma, EmbeddingGemma, etc.) |
|
||||
|
||||
|
||||
@@ -0,0 +1,128 @@
|
||||
# DiffusionGemma — Research + Smoke Test (2026-06-17)
|
||||
|
||||
> Read when: scoping a text-diffusion model, deciding whether DiffusionGemma is
|
||||
> deployable on the homelab, reproducing the llama-diffusion-cli build, or
|
||||
> comparing diffusion vs autoregressive Gemma 4 throughput.
|
||||
|
||||
## What it is
|
||||
|
||||
`google/diffusiongemma-26B-A4B-it` — released **2026-06-10**, Apache-2.0. Google
|
||||
DeepMind's first open-weight **text-diffusion** LLM. Instead of autoregressive
|
||||
token-by-token decoding, it **denoises a 256-token "canvas" in parallel** over a
|
||||
handful of iterative steps (block-autoregressive multi-canvas sampling).
|
||||
|
||||
| Property | Value |
|
||||
|---|---|
|
||||
| Base | Gemma 4 26B MoE — 25.2B total / **3.8B active** (8 of 128 experts), 30 layers |
|
||||
| Generation | Discrete text diffusion, 256-token canvas, entropy-bound decoder |
|
||||
| Modalities | text + image + video in; **text out** (no audio) |
|
||||
| Context | up to 256K |
|
||||
| Knowledge cutoff | Jan 2025 |
|
||||
| Speed claim | 1,100+ tok/s on H100 FP8; "up to 4×" on constrained consumer HW |
|
||||
| New TF class | `DiffusionGemmaForBlockDiffusion` |
|
||||
| Day-zero serving | vLLM, Transformers, MLX, SGLang |
|
||||
|
||||
## The deployment snag: it does NOT run in your Ollama stack
|
||||
|
||||
Stock `ollama` / `llama-cli` / `llama-server` reject it: **`unknown model
|
||||
architecture: 'diffusion-gemma'`**. Ollama issue #16664 is an unmerged feature
|
||||
request. It needs the dedicated **`llama-diffusion-cli`** binary from
|
||||
**ggml-org/llama.cpp PR #24423** (danielhanchen / Unsloth), which adds the
|
||||
`diffusion-gemma` arch + the entropy-bound canvas decoder. The standard runners
|
||||
cannot generate from it even with that PR — only `llama-diffusion-cli` /
|
||||
`diffusion-gemma-server` can.
|
||||
|
||||
GGUF quants (`unsloth/diffusiongemma-26B-A4B-it-GGUF`): Q4_K_M 16.8 GB · Q5_K_M
|
||||
19.1 · Q6_K 22.7 · Q8_0 26.9 · BF16 50.5.
|
||||
|
||||
## Build recipe (reproduced on steel141, 2026-06-17)
|
||||
|
||||
Prereqs that were **missing** on steel141 and had to be installed:
|
||||
- **CUDA toolkit** (`nvcc`) — none present (Ollama ships runtime libs, not the
|
||||
compiler). Installed `nvidia-cuda-toolkit` 12.4.131 via apt; matches driver
|
||||
CUDA 12.4. ~2.5 GB.
|
||||
- **`gh`** — absent. Used `git fetch origin pull/24423/head:pr-24423` instead.
|
||||
|
||||
```bash
|
||||
# on /mnt/ai_data (ollama-owned; sudo mkdir + chown claude first — 325 GB free there)
|
||||
git clone --depth 1 https://github.com/ggml-org/llama.cpp.git
|
||||
cd llama.cpp
|
||||
git fetch --depth 1 origin pull/24423/head:pr-24423 && git checkout pr-24423
|
||||
cmake -B build -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES="86" -DLLAMA_CURL=OFF -DCMAKE_BUILD_TYPE=Release
|
||||
cmake --build build -j 24 --target llama-diffusion-cli
|
||||
# binary: build/bin/llama-diffusion-cli (sm_86 = 3090 Ti; add 75 for the 1660)
|
||||
|
||||
hf download unsloth/diffusiongemma-26B-A4B-it-GGUF \
|
||||
diffusiongemma-26B-A4B-it-Q4_K_M.gguf --local-dir /mnt/ai_data/diffusiongemma/gguf
|
||||
```
|
||||
|
||||
Local artifacts live at `/mnt/ai_data/diffusiongemma/` (GGUF + llama.cpp build).
|
||||
Harness: `scripts/diffusiongemma-smoketest/run.sh`; raw logs in `results/`.
|
||||
|
||||
### GPU-selection gotcha (cost me two failed runs)
|
||||
|
||||
**nvidia-smi index ≠ CUDA index on steel141.** nvidia-smi lists the 3090 Ti as
|
||||
GPU **1** and the 1660 SUPER as GPU 0; CUDA enumerates them the *other way*
|
||||
(3090 Ti = CUDA device **0**). So `CUDA_VISIBLE_DEVICES=1` selected the 6 GB
|
||||
1660 → `cudaMalloc 16013 MiB ... out of memory`. Fix: **select by UUID** —
|
||||
`CUDA_VISIBLE_DEVICES=GPU-61fed72d-...` (order-independent), or set
|
||||
`CUDA_DEVICE_ORDER=PCI_BUS_ID` and verify. Get the UUID from `nvidia-smi -L`.
|
||||
|
||||
Also: the 16 GB model won't fit alongside a loaded `gemma4:26b` (it splits
|
||||
~14 GB onto the 3090 Ti). For each run I **stopped ollama** (`sudo systemctl
|
||||
stop ollama`), ran, then restarted it — `ollama stop <model>` alone loses the
|
||||
race because live traffic re-loads the model within minutes. Bounded ~3-min
|
||||
windows; ollama recovered clean each time.
|
||||
|
||||
## Smoke-test results (Q4_K_M, 3090 Ti, `--diffusion-eb auto`, `-n 256`)
|
||||
|
||||
| Prompt | Steps/block | ms/step | Effective tok/s | In-step parallel | Verdict |
|
||||
|---|---|---|---|---|---|
|
||||
| reasoning (3 sentences) | 19 | 126 | 106.8 | 2030 tok/s | coherent, on-topic |
|
||||
| code (`is_prime` + docstring) | 13 | 135 | **146.2** | 1900 tok/s | **correct Python**, right algorithm |
|
||||
| format (5 colors, 1 block) | 21 | 124 | 98.6 | 2070 tok/s | truncated mid-thought (see below) |
|
||||
| format (4 blocks, `-n 512`) | 25.5 avg / 2 blocks | 124 | 81.1 | 2068 tok/s | still in thought channel, no final answer |
|
||||
|
||||
- **Model load:** ~7 s (16 GB weights → 3090 Ti). VRAM ~16 GB.
|
||||
- **Diffusion confirmed:** the entropy-bound decoder converges a 256-token canvas
|
||||
in **13–25 steps** (max 48), ~124 ms/step. "In-step parallel" ≈ **1,900–2,070
|
||||
tok/s** is the canvas-parallel rate; **effective 81–146 tok/s** is what you
|
||||
actually get end-to-end. For comparison the AR `gemma4:26b` MoE does ~134 tok/s
|
||||
— DiffusionGemma is in the same ballpark via a totally different mechanism, and
|
||||
would scale better as canvases lengthen.
|
||||
|
||||
### Behavioral finding: the `<|channel>thought` channel eats the canvas
|
||||
|
||||
DiffusionGemma-**it** emits an explicit chain-of-thought block
|
||||
(`<|channel>thought ...`) that is denoised *inside* the canvas. On a single
|
||||
256-token canvas, prompts that trigger long deliberation get **truncated before
|
||||
the final answer**. The "5 primary colors of light" prompt is mildly adversarial
|
||||
(there are only *three*), and the model spiraled — even at 4 blocks / 512 tokens
|
||||
it was still weighing whether five primaries exist and never emitted a list. The
|
||||
reasoning and code prompts, which don't fight the model, came out clean.
|
||||
|
||||
**Takeaway for anyone building on it:** budget canvas blocks for *thought +
|
||||
answer*, not just answer (`--diffusion-blocks` ≥ 4 for non-trivial prompts), or
|
||||
steer/suppress the thought channel. Don't expect tight format adherence in a
|
||||
single canvas.
|
||||
|
||||
## Bottom line
|
||||
|
||||
- DiffusionGemma is **real, runs locally on a single 3090 Ti** (Q4_K_M, ~16 GB)
|
||||
via the PR #24423 `llama-diffusion-cli`, and the diffusion mechanism works.
|
||||
- It is **not deployable through your Ollama stack** today and won't be until the
|
||||
diffusion arch merges into llama.cpp mainline (then Ollama). For now it's an
|
||||
**experimental research artifact**, not a drop-in for Simon / AI_Visualizer /
|
||||
mort-bot.
|
||||
- Generation quality at Q4_K_M is solid on cooperative prompts (correct code,
|
||||
coherent reasoning) but the thought-channel-eats-canvas behavior makes strict
|
||||
short-format outputs unreliable without block budgeting.
|
||||
|
||||
## Sources
|
||||
|
||||
- https://ai.google.dev/gemma/docs/diffusiongemma
|
||||
- https://huggingface.co/google/diffusiongemma-26B-A4B-it
|
||||
- https://huggingface.co/unsloth/diffusiongemma-26B-A4B-it-GGUF
|
||||
- https://github.com/ggml-org/llama.cpp (PR #24423)
|
||||
- https://github.com/ollama/ollama/issues/16664
|
||||
- https://www.theregister.com/ai-and-ml/2026/06/11/googles-diffusiongemma-uses-diffusion-tech-to-speed-text-generation/
|
||||
@@ -0,0 +1,44 @@
|
||||
0.00.836.466 W load: control-looking token: 212 '</s>' was not control-type; this is probably a bug in the model. its type will be overridden
|
||||
0.00.837.141 W load: control-looking token: 50 '<|tool_response>' was not control-type; this is probably a bug in the model. its type will be overridden
|
||||
0.00.873.644 W load: special_eog_ids contains '<|tool_response>', removing '</s>' token from EOG list
|
||||
0.04.996.883 I diffusion: -n 160 -> 1 blocks, n_ubatch=2304 n_batch=2304 n_ctx=2304 (canvas_length=256)
|
||||
0.04.996.886 I diffusion: --fit has no effect here; context is sized from -n and the canvas. Set -ngl / --n-cpu-moe to control device memory.
|
||||
0.04.997.441 W llama_context: n_ctx_seq (2304) < n_ctx_train (262144) -- the full capacity of the model will not be utilized
|
||||
0.05.002.413 W sched_reserve: layer 5 is assigned to device CUDA0 but the Flash Attention tensor is assigned to device CPU (usually due to missing support)
|
||||
0.05.002.416 W sched_reserve: Flash Attention was auto, set to disabled
|
||||
0.07.343.745 I diffusion_params: steps=128 schedule=0 algorithm=4 temperature=0.800 eps=0.001000 mask_token=4
|
||||
0.07.344.014 I diffusion_eb: max_steps=48 t=[0.400,0.800] entropy_bound=0.1000 stability=1 confidence=0.0050 kv_cache=on gpu_sampling=on sample_reduce=on
|
||||
0.07.345.228 W init: embeddings required but some input tokens were not marked as outputs -> overriding
|
||||
0.07.822.959 I
|
||||
diffusion step: 0/48 [ ] 0%0.07.925.523 I
|
||||
diffusion step: 1/48 [= ] 2%0.08.028.933 I
|
||||
diffusion step: 2/48 [== ] 4%0.08.132.168 I
|
||||
diffusion step: 3/48 [=== ] 6%0.08.238.755 I
|
||||
diffusion step: 4/48 [==== ] 8%0.08.343.911 I
|
||||
diffusion step: 5/48 [===== ] 10%0.08.450.845 I
|
||||
diffusion step: 6/48 [====== ] 12%0.08.558.796 I
|
||||
diffusion step: 7/48 [======= ] 14%0.08.665.123 I
|
||||
diffusion step: 8/48 [======== ] 16%0.08.771.930 I
|
||||
diffusion step: 9/48 [========= ] 18%0.08.881.776 I
|
||||
diffusion step: 10/48 [========== ] 20%0.08.988.729 I
|
||||
diffusion step: 11/48 [=========== ] 22%0.09.096.269 I
|
||||
diffusion step: 12/48 [============ ] 25%
|
||||
<|channel>thought
|
||||
* Task: Write a Python function `is_prime(n)`.
|
||||
* Requirement: Include a docstring.
|
||||
* Constraint: Output *only* the code, no explanation.
|
||||
|
||||
* A prime number is a natural number greater than 1 that has no positive divisors other than 1 and itself.
|
||||
* If $n \le 1$, not prime.
|
||||
* If $n = 2$, prime.
|
||||
* If $n$ is even and $>2$, not prime.
|
||||
* Check divisors from 3 to $\sqrt{n}$.
|
||||
|
||||
```python
|
||||
def is_prime(n):
|
||||
"""
|
||||
Determines if a number n is prime.
|
||||
|
||||
Args:
|
||||
n (int): The number to check.
|
||||
|
||||
@@ -0,0 +1,31 @@
|
||||
0.00.841.557 W load: control-looking token: 212 '</s>' was not control-type; this is probably a bug in the model. its type will be overridden
|
||||
0.00.842.233 W load: control-looking token: 50 '<|tool_response>' was not control-type; this is probably a bug in the model. its type will be overridden
|
||||
0.00.879.927 W load: special_eog_ids contains '<|tool_response>', removing '</s>' token from EOG list
|
||||
0.04.814.391 I diffusion: -n 48 -> 1 blocks, n_ubatch=2304 n_batch=2304 n_ctx=2304 (canvas_length=256)
|
||||
0.04.814.394 I diffusion: --fit has no effect here; context is sized from -n and the canvas. Set -ngl / --n-cpu-moe to control device memory.
|
||||
0.04.814.907 W llama_context: n_ctx_seq (2304) < n_ctx_train (262144) -- the full capacity of the model will not be utilized
|
||||
0.04.819.412 W sched_reserve: layer 5 is assigned to device CUDA0 but the Flash Attention tensor is assigned to device CPU (usually due to missing support)
|
||||
0.04.819.415 W sched_reserve: Flash Attention was auto, set to disabled
|
||||
0.07.106.365 I diffusion_params: steps=128 schedule=0 algorithm=4 temperature=0.800 eps=0.001000 mask_token=4
|
||||
0.07.106.635 I diffusion_eb: max_steps=48 t=[0.400,0.800] entropy_bound=0.1000 stability=1 confidence=0.0050 kv_cache=on gpu_sampling=on sample_reduce=on
|
||||
0.07.107.839 W init: embeddings required but some input tokens were not marked as outputs -> overriding
|
||||
0.07.565.506 I
|
||||
diffusion step: 0/48 [ ] 0%0.07.667.965 I
|
||||
diffusion step: 1/48 [= ] 2%0.07.772.514 I
|
||||
diffusion step: 2/48 [== ] 4%0.07.876.431 I
|
||||
diffusion step: 3/48 [=== ] 6%0.07.981.552 I
|
||||
diffusion step: 4/48 [==== ] 8%0.08.087.446 I
|
||||
diffusion step: 5/48 [===== ] 10%0.08.196.277 I
|
||||
diffusion step: 6/48 [====== ] 12%0.08.302.551 I
|
||||
diffusion step: 7/48 [======= ] 14%0.08.407.803 I
|
||||
diffusion step: 8/48 [======== ] 16%0.08.514.697 I
|
||||
diffusion step: 9/48 [========= ] 18%0.08.621.268 I
|
||||
diffusion step: 10/48 [========== ] 20%0.08.728.440 I
|
||||
diffusion step: 11/48 [=========== ] 22%0.08.835.336 I
|
||||
diffusion step: 12/48 [============ ] 25%0.08.942.227 I
|
||||
diffusion step: 13/48 [============= ] 27%0.09.050.835 I
|
||||
diffusion step: 14/48 [============== ] 29%0.09.160.522 I
|
||||
diffusion step: 15/48 [=============== ] 31%0.09.268.593 I
|
||||
diffusion step: 16/48 [================ ] 33%0.09.377.200 I
|
||||
diffusion step: 17/48 [================= ] 35%0.09.485.784 I
|
||||
diffusion step: 18/48 [================== ] 37%0.09.595.474 I
|
||||
@@ -0,0 +1,43 @@
|
||||
0.00.778.413 W load: control-looking token: 212 '</s>' was not control-type; this is probably a bug in the model. its type will be overridden
|
||||
0.00.778.868 W load: control-looking token: 50 '<|tool_response>' was not control-type; this is probably a bug in the model. its type will be overridden
|
||||
0.00.803.531 W load: special_eog_ids contains '<|tool_response>', removing '</s>' token from EOG list
|
||||
0.04.672.736 I diffusion: -n 512 -> 2 blocks, n_ubatch=2560 n_batch=2560 n_ctx=2560 (canvas_length=256)
|
||||
0.04.672.739 I diffusion: --fit has no effect here; context is sized from -n and the canvas. Set -ngl / --n-cpu-moe to control device memory.
|
||||
0.04.673.245 W llama_context: n_ctx_seq (2560) < n_ctx_train (262144) -- the full capacity of the model will not be utilized
|
||||
0.04.677.664 W sched_reserve: layer 5 is assigned to device CUDA0 but the Flash Attention tensor is assigned to device CPU (usually due to missing support)
|
||||
0.04.677.667 W sched_reserve: Flash Attention was auto, set to disabled
|
||||
0.06.828.575 I diffusion_params: steps=128 schedule=0 algorithm=4 temperature=0.800 eps=0.001000 mask_token=4
|
||||
0.06.828.825 I diffusion_eb: max_steps=48 t=[0.400,0.800] entropy_bound=0.1000 stability=1 confidence=0.0050 kv_cache=on gpu_sampling=on sample_reduce=on
|
||||
0.06.829.987 W init: embeddings required but some input tokens were not marked as outputs -> overriding
|
||||
0.07.297.536 I
|
||||
diffusion step: 0/48 [ ] 0%0.07.401.754 I
|
||||
diffusion step: 1/48 [= ] 2%0.07.506.767 I
|
||||
diffusion step: 2/48 [== ] 4%0.07.612.593 I
|
||||
diffusion step: 3/48 [=== ] 6%0.07.720.060 I
|
||||
diffusion step: 4/48 [==== ] 8%0.07.827.278 I
|
||||
diffusion step: 5/48 [===== ] 10%0.07.933.928 I
|
||||
diffusion step: 6/48 [====== ] 12%0.08.042.088 I
|
||||
diffusion step: 7/48 [======= ] 14%0.08.150.433 I
|
||||
diffusion step: 8/48 [======== ] 16%0.08.259.575 I
|
||||
diffusion step: 9/48 [========= ] 18%0.08.367.825 I
|
||||
diffusion step: 10/48 [========== ] 20%0.08.475.147 I
|
||||
diffusion step: 11/48 [=========== ] 22%0.08.584.038 I
|
||||
diffusion step: 12/48 [============ ] 25%0.08.693.905 I
|
||||
diffusion step: 13/48 [============= ] 27%0.08.803.010 I
|
||||
diffusion step: 14/48 [============== ] 29%0.08.912.275 I
|
||||
diffusion step: 15/48 [=============== ] 31%0.09.021.574 I
|
||||
diffusion step: 16/48 [================ ] 33%0.09.130.837 I
|
||||
diffusion step: 17/48 [================= ] 35%0.09.240.344 I
|
||||
diffusion step: 18/48 [================== ] 37%0.09.350.482 I
|
||||
diffusion step: 19/48 [=================== ] 39%0.09.458.691 I
|
||||
diffusion step: 20/48 [==================== ] 41%0.09.458.811 W init: embeddings required but some input tokens were not marked as outputs -> overriding
|
||||
0.09.908.398 I
|
||||
diffusion step: 0/48 [ ] 0%0.10.012.788 I
|
||||
diffusion step: 1/48 [= ] 2%0.10.116.823 I
|
||||
diffusion step: 2/48 [== ] 4%0.10.221.634 I
|
||||
diffusion step: 3/48 [=== ] 6%0.10.327.638 I
|
||||
diffusion step: 4/48 [==== ] 8%0.10.436.834 I
|
||||
diffusion step: 5/48 [===== ] 10%0.10.544.714 I
|
||||
diffusion step: 6/48 [====== ] 12%0.10.652.512 I
|
||||
diffusion step: 7/48 [======= ] 14%0.10.762.174 I
|
||||
diffusion step: 8/48 [======== ] 16%0.10.871.902 I
|
||||
@@ -0,0 +1,27 @@
|
||||
0.00.896.419 W load: control-looking token: 212 '</s>' was not control-type; this is probably a bug in the model. its type will be overridden
|
||||
0.00.897.085 W load: control-looking token: 50 '<|tool_response>' was not control-type; this is probably a bug in the model. its type will be overridden
|
||||
0.00.933.523 W load: special_eog_ids contains '<|tool_response>', removing '</s>' token from EOG list
|
||||
0.04.856.194 I diffusion: -n 128 -> 1 blocks, n_ubatch=2304 n_batch=2304 n_ctx=2304 (canvas_length=256)
|
||||
0.04.856.197 I diffusion: --fit has no effect here; context is sized from -n and the canvas. Set -ngl / --n-cpu-moe to control device memory.
|
||||
0.04.864.961 W llama_context: n_ctx_seq (2304) < n_ctx_train (262144) -- the full capacity of the model will not be utilized
|
||||
0.04.869.425 W sched_reserve: layer 5 is assigned to device CUDA0 but the Flash Attention tensor is assigned to device CPU (usually due to missing support)
|
||||
0.04.869.427 W sched_reserve: Flash Attention was auto, set to disabled
|
||||
0.07.226.382 I diffusion_params: steps=128 schedule=0 algorithm=4 temperature=0.800 eps=0.001000 mask_token=4
|
||||
0.07.226.626 I diffusion_eb: max_steps=48 t=[0.400,0.800] entropy_bound=0.1000 stability=1 confidence=0.0050 kv_cache=on gpu_sampling=on sample_reduce=on
|
||||
0.07.228.370 W init: embeddings required but some input tokens were not marked as outputs -> overriding
|
||||
0.07.721.777 I
|
||||
diffusion step: 0/48 [ ] 0%0.07.822.673 I
|
||||
diffusion step: 1/48 [= ] 2%0.07.924.966 I
|
||||
diffusion step: 2/48 [== ] 4%0.08.030.579 I
|
||||
diffusion step: 3/48 [=== ] 6%0.08.135.224 I
|
||||
diffusion step: 4/48 [==== ] 8%0.08.240.214 I
|
||||
diffusion step: 5/48 [===== ] 10%0.08.346.392 I
|
||||
diffusion step: 6/48 [====== ] 12%0.08.452.589 I
|
||||
diffusion step: 7/48 [======= ] 14%0.08.556.808 I
|
||||
diffusion step: 8/48 [======== ] 16%0.08.663.759 I
|
||||
diffusion step: 9/48 [========= ] 18%0.08.770.846 I
|
||||
diffusion step: 10/48 [========== ] 20%0.08.876.181 I
|
||||
diffusion step: 11/48 [=========== ] 22%0.08.984.389 I
|
||||
diffusion step: 12/48 [============ ] 25%0.09.090.492 I
|
||||
diffusion step: 13/48 [============= ] 27%0.09.198.211 I
|
||||
diffusion step: 14/48 [============== ] 29%0.09.304.460 I
|
||||
Executable
+46
@@ -0,0 +1,46 @@
|
||||
#!/usr/bin/env bash
|
||||
# DiffusionGemma smoke test — steel141 3090 Ti
|
||||
#
|
||||
# DiffusionGemma (google/diffusiongemma-26B-A4B-it, released 2026-06-10) is a
|
||||
# text-DIFFUSION MoE model. It does NOT run in Ollama or stock llama.cpp:
|
||||
# the standard llama-cli/llama-server reject arch 'diffusion-gemma'. It needs
|
||||
# the dedicated `llama-diffusion-cli` binary from ggml-org/llama.cpp PR #24423
|
||||
# (danielhanchen / Unsloth), which denoises 256-token canvas blocks in parallel.
|
||||
#
|
||||
# This harness runs a few non-interactive prompts and captures wall-clock + the
|
||||
# diffusion step/block telemetry the CLI prints.
|
||||
set -euo pipefail
|
||||
|
||||
CLI="${CLI:-/mnt/ai_data/diffusiongemma/llama.cpp/build/bin/llama-diffusion-cli}"
|
||||
MODEL="${MODEL:-/mnt/ai_data/diffusiongemma/gguf/diffusiongemma-26B-A4B-it-Q4_K_M.gguf}"
|
||||
NGL="${NGL:-99}" # offload all layers to GPU
|
||||
NPRED="${NPRED:-256}" # one full canvas
|
||||
OUTDIR="${OUTDIR:-$(dirname "$0")/results}"
|
||||
mkdir -p "$OUTDIR"
|
||||
|
||||
[ -x "$CLI" ] || { echo "missing CLI: $CLI" >&2; exit 1; }
|
||||
[ -f "$MODEL" ] || { echo "missing model: $MODEL" >&2; exit 1; }
|
||||
|
||||
run() {
|
||||
local name="$1"; local prompt="$2"; shift 2
|
||||
local log="$OUTDIR/${name}.log"
|
||||
echo "=== $name ===" | tee "$log"
|
||||
echo "prompt: $prompt" | tee -a "$log"
|
||||
local t0 t1
|
||||
t0=$(date +%s.%N)
|
||||
"$CLI" -m "$MODEL" -ngl "$NGL" -n "$NPRED" --diffusion-eb auto -p "$prompt" "$@" 2>&1 | tee -a "$log"
|
||||
t1=$(date +%s.%N)
|
||||
echo "WALL_SECONDS=$(echo "$t1 - $t0" | bc)" | tee -a "$log"
|
||||
echo | tee -a "$log"
|
||||
}
|
||||
|
||||
# 1) Plain reasoning — sanity + coherence
|
||||
run reasoning "Explain in three sentences why diffusion language models can be faster than autoregressive ones."
|
||||
|
||||
# 2) Code — structured output the diffusion canvas has to fill coherently
|
||||
run code "Write a Python function is_prime(n) with a docstring. Output only the code."
|
||||
|
||||
# 3) Instruction following with a hard format constraint
|
||||
run format "List exactly five primary colors of light, one per line, no extra text."
|
||||
|
||||
echo "All runs complete. Logs in $OUTDIR"
|
||||
Reference in New Issue
Block a user