diff --git a/CORPUS_ollama_variants.md b/CORPUS_ollama_variants.md index bce90c3..6aaea96 100644 --- a/CORPUS_ollama_variants.md +++ b/CORPUS_ollama_variants.md @@ -15,6 +15,8 @@ > **⚠️ Ollama 0.20.4 pin invalidated (2026-06-16):** steel141 was upgraded 0.20.4 → 0.30.8 to pull `gemma4:12b`. The `mort-bakeoff-2026-04-18.md` finding ("thinking does NOT accumulate in context **on Ollama 0.20.4**") was pinned to that version and must be **re-verified on 0.30.8** before relying on it. Binary backup + override saved in `.backup/ollama-bin-0.20.4-*`. +> **DiffusionGemma is NOT an Ollama variant (do not try to pull it).** `google/diffusiongemma-26B-A4B-it` (released 2026-06-10, the 26B-A4B MoE backbone + a text-**diffusion** head) is a separate generation paradigm — it denoises a 256-token canvas in parallel rather than decoding token-by-token. Ollama/`llama-cli`/`llama-server` reject it (`unknown model architecture: 'diffusion-gemma'`); it needs the dedicated `llama-diffusion-cli` from **ggml-org/llama.cpp PR #24423**. Smoke-tested locally on steel141's 3090 Ti 2026-06-17 (Q4_K_M, ~16 GB, ~106 tok/s effective / ~2030 tok/s in-step-parallel; correct code, coherent reasoning; the `<|channel>thought` CoT eats the canvas so strict short formats need block budgeting). Experimental research artifact, **not deployable through the homelab Ollama stack** until the diffusion arch merges to llama.cpp mainline. Full writeup + build recipe: `docs/reference/diffusiongemma-smoketest-2026-06-17.md`. + ## Capabilities by Variant (from `ollama show`) All variants support: diff --git a/README.md b/README.md index 69956ed..d27a224 100644 --- a/README.md +++ b/README.md @@ -20,6 +20,7 @@ Research corpus and implementation guidance for Google Gemma 4, based on product | `docs/openwebui-setup.md` | How to configure Gemma 4 inside OpenWebUI — per-setting reference, two ready-to-bake Workspace Model profiles (chat + extract), and a symptom→cause troubleshooting table mapped back to GOTCHAS.md. Assumes Ollama + OpenWebUI are already running. | When setting up or debugging a Gemma 4 model in OpenWebUI, or handing the front-end config to someone else | | `docs/reference/bakeoff-2026-04-18.md` | CLI-coding-agent bakeoff on 3090 Ti. **Rounds 1/2 misidentified the cause; Round 3 (the correct one): `think: false` silent-stops gemma4:26b at certain multi-turn states on 32K context.** 31B and Qwen3-Coder robust to the flag. Harness at `scripts/bakeoff/` | When deciding which model to back a CLI agent with, writing a custom agent payload, or debugging a silent tool-call halt | | `docs/reference/mort-bakeoff-2026-04-18.md` | mort-bot-specific `think=true` vs `think=false` bakeoff on mort's actual loop shape (gemma4:26b, num_ctx=8192). **Thinking does NOT accumulate in context on Ollama 0.20.4** — strips it from serialized history. Both settings behave identically on step counts, tool counts, wall clock. Harness at `scripts/mort-bakeoff/` | When deciding mort-bot's THINK env var, or when someone claims "think=true eats context" without pinning an Ollama version | +| `docs/reference/diffusiongemma-smoketest-2026-06-17.md` | **DiffusionGemma** (`google/diffusiongemma-26B-A4B-it`, released 2026-06-10) — Google's first open-weight text-**diffusion** LLM (26B-A4B MoE + canvas diffusion head). Research + first-hand smoke test on steel141's 3090 Ti: does NOT run in Ollama (`unknown model architecture: 'diffusion-gemma'`), needs `llama-diffusion-cli` from ggml-org/llama.cpp PR #24423. Build recipe, throughput (~106 tok/s effective / ~2030 tok/s in-step-parallel at Q4_K_M), the nvidia-smi-vs-CUDA device-ordering gotcha, and the "thought channel eats the canvas" behavior. | When evaluating a text-diffusion model, deciding if DiffusionGemma is homelab-deployable, or reproducing the llama-diffusion-cli build | | `docs/reference/gpu-bakeoff-2026-04-20.md` | Cross-GPU throughput bakeoff: steel141 RTX 3090 Ti vs strix-halo (AMD Strix Halo). **3090 Ti wins decode decisively (128 tok/s on 26B MoE). Strix gets ~42% of that on ~25% of the bandwidth.** Also quantifies the MoE vs dense gap: 26B decodes ~4.7× faster than 31B on both cards. Harness at `scripts/gpu-bakeoff/` | When choosing which host to run a Gemma 4 workload on | | `tooling/` | **Canonical upstream tooling** — real scripts, notebooks, model cards, and configs pulled from Google / HF / framework maintainers (147 files). Subdirs: `google-official/`, `huggingface/`, `inference-frameworks/`, `gemma-family/`, `fine-tuning/`. See `tooling/README.md` for index and findings that update the older `CORPUS_*` docs | When you need authoritative source material — model cards, chat templates, fine-tuning recipes, serving commands for vLLM / llama.cpp / MLX, or to scope a specialized sibling (ShieldGemma, EmbeddingGemma, etc.) | diff --git a/docs/reference/diffusiongemma-smoketest-2026-06-17.md b/docs/reference/diffusiongemma-smoketest-2026-06-17.md new file mode 100644 index 0000000..f3e7c70 --- /dev/null +++ b/docs/reference/diffusiongemma-smoketest-2026-06-17.md @@ -0,0 +1,128 @@ +# DiffusionGemma — Research + Smoke Test (2026-06-17) + +> Read when: scoping a text-diffusion model, deciding whether DiffusionGemma is +> deployable on the homelab, reproducing the llama-diffusion-cli build, or +> comparing diffusion vs autoregressive Gemma 4 throughput. + +## What it is + +`google/diffusiongemma-26B-A4B-it` — released **2026-06-10**, Apache-2.0. Google +DeepMind's first open-weight **text-diffusion** LLM. Instead of autoregressive +token-by-token decoding, it **denoises a 256-token "canvas" in parallel** over a +handful of iterative steps (block-autoregressive multi-canvas sampling). + +| Property | Value | +|---|---| +| Base | Gemma 4 26B MoE — 25.2B total / **3.8B active** (8 of 128 experts), 30 layers | +| Generation | Discrete text diffusion, 256-token canvas, entropy-bound decoder | +| Modalities | text + image + video in; **text out** (no audio) | +| Context | up to 256K | +| Knowledge cutoff | Jan 2025 | +| Speed claim | 1,100+ tok/s on H100 FP8; "up to 4×" on constrained consumer HW | +| New TF class | `DiffusionGemmaForBlockDiffusion` | +| Day-zero serving | vLLM, Transformers, MLX, SGLang | + +## The deployment snag: it does NOT run in your Ollama stack + +Stock `ollama` / `llama-cli` / `llama-server` reject it: **`unknown model +architecture: 'diffusion-gemma'`**. Ollama issue #16664 is an unmerged feature +request. It needs the dedicated **`llama-diffusion-cli`** binary from +**ggml-org/llama.cpp PR #24423** (danielhanchen / Unsloth), which adds the +`diffusion-gemma` arch + the entropy-bound canvas decoder. The standard runners +cannot generate from it even with that PR — only `llama-diffusion-cli` / +`diffusion-gemma-server` can. + +GGUF quants (`unsloth/diffusiongemma-26B-A4B-it-GGUF`): Q4_K_M 16.8 GB · Q5_K_M +19.1 · Q6_K 22.7 · Q8_0 26.9 · BF16 50.5. + +## Build recipe (reproduced on steel141, 2026-06-17) + +Prereqs that were **missing** on steel141 and had to be installed: +- **CUDA toolkit** (`nvcc`) — none present (Ollama ships runtime libs, not the + compiler). Installed `nvidia-cuda-toolkit` 12.4.131 via apt; matches driver + CUDA 12.4. ~2.5 GB. +- **`gh`** — absent. Used `git fetch origin pull/24423/head:pr-24423` instead. + +```bash +# on /mnt/ai_data (ollama-owned; sudo mkdir + chown claude first — 325 GB free there) +git clone --depth 1 https://github.com/ggml-org/llama.cpp.git +cd llama.cpp +git fetch --depth 1 origin pull/24423/head:pr-24423 && git checkout pr-24423 +cmake -B build -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES="86" -DLLAMA_CURL=OFF -DCMAKE_BUILD_TYPE=Release +cmake --build build -j 24 --target llama-diffusion-cli +# binary: build/bin/llama-diffusion-cli (sm_86 = 3090 Ti; add 75 for the 1660) + +hf download unsloth/diffusiongemma-26B-A4B-it-GGUF \ + diffusiongemma-26B-A4B-it-Q4_K_M.gguf --local-dir /mnt/ai_data/diffusiongemma/gguf +``` + +Local artifacts live at `/mnt/ai_data/diffusiongemma/` (GGUF + llama.cpp build). +Harness: `scripts/diffusiongemma-smoketest/run.sh`; raw logs in `results/`. + +### GPU-selection gotcha (cost me two failed runs) + +**nvidia-smi index ≠ CUDA index on steel141.** nvidia-smi lists the 3090 Ti as +GPU **1** and the 1660 SUPER as GPU 0; CUDA enumerates them the *other way* +(3090 Ti = CUDA device **0**). So `CUDA_VISIBLE_DEVICES=1` selected the 6 GB +1660 → `cudaMalloc 16013 MiB ... out of memory`. Fix: **select by UUID** — +`CUDA_VISIBLE_DEVICES=GPU-61fed72d-...` (order-independent), or set +`CUDA_DEVICE_ORDER=PCI_BUS_ID` and verify. Get the UUID from `nvidia-smi -L`. + +Also: the 16 GB model won't fit alongside a loaded `gemma4:26b` (it splits +~14 GB onto the 3090 Ti). For each run I **stopped ollama** (`sudo systemctl +stop ollama`), ran, then restarted it — `ollama stop ` alone loses the +race because live traffic re-loads the model within minutes. Bounded ~3-min +windows; ollama recovered clean each time. + +## Smoke-test results (Q4_K_M, 3090 Ti, `--diffusion-eb auto`, `-n 256`) + +| Prompt | Steps/block | ms/step | Effective tok/s | In-step parallel | Verdict | +|---|---|---|---|---|---| +| reasoning (3 sentences) | 19 | 126 | 106.8 | 2030 tok/s | coherent, on-topic | +| code (`is_prime` + docstring) | 13 | 135 | **146.2** | 1900 tok/s | **correct Python**, right algorithm | +| format (5 colors, 1 block) | 21 | 124 | 98.6 | 2070 tok/s | truncated mid-thought (see below) | +| format (4 blocks, `-n 512`) | 25.5 avg / 2 blocks | 124 | 81.1 | 2068 tok/s | still in thought channel, no final answer | + +- **Model load:** ~7 s (16 GB weights → 3090 Ti). VRAM ~16 GB. +- **Diffusion confirmed:** the entropy-bound decoder converges a 256-token canvas + in **13–25 steps** (max 48), ~124 ms/step. "In-step parallel" ≈ **1,900–2,070 + tok/s** is the canvas-parallel rate; **effective 81–146 tok/s** is what you + actually get end-to-end. For comparison the AR `gemma4:26b` MoE does ~134 tok/s + — DiffusionGemma is in the same ballpark via a totally different mechanism, and + would scale better as canvases lengthen. + +### Behavioral finding: the `<|channel>thought` channel eats the canvas + +DiffusionGemma-**it** emits an explicit chain-of-thought block +(`<|channel>thought ...`) that is denoised *inside* the canvas. On a single +256-token canvas, prompts that trigger long deliberation get **truncated before +the final answer**. The "5 primary colors of light" prompt is mildly adversarial +(there are only *three*), and the model spiraled — even at 4 blocks / 512 tokens +it was still weighing whether five primaries exist and never emitted a list. The +reasoning and code prompts, which don't fight the model, came out clean. + +**Takeaway for anyone building on it:** budget canvas blocks for *thought + +answer*, not just answer (`--diffusion-blocks` ≥ 4 for non-trivial prompts), or +steer/suppress the thought channel. Don't expect tight format adherence in a +single canvas. + +## Bottom line + +- DiffusionGemma is **real, runs locally on a single 3090 Ti** (Q4_K_M, ~16 GB) + via the PR #24423 `llama-diffusion-cli`, and the diffusion mechanism works. +- It is **not deployable through your Ollama stack** today and won't be until the + diffusion arch merges into llama.cpp mainline (then Ollama). For now it's an + **experimental research artifact**, not a drop-in for Simon / AI_Visualizer / + mort-bot. +- Generation quality at Q4_K_M is solid on cooperative prompts (correct code, + coherent reasoning) but the thought-channel-eats-canvas behavior makes strict + short-format outputs unreliable without block budgeting. + +## Sources + +- https://ai.google.dev/gemma/docs/diffusiongemma +- https://huggingface.co/google/diffusiongemma-26B-A4B-it +- https://huggingface.co/unsloth/diffusiongemma-26B-A4B-it-GGUF +- https://github.com/ggml-org/llama.cpp (PR #24423) +- https://github.com/ollama/ollama/issues/16664 +- https://www.theregister.com/ai-and-ml/2026/06/11/googles-diffusiongemma-uses-diffusion-tech-to-speed-text-generation/ diff --git a/scripts/diffusiongemma-smoketest/results/code.log b/scripts/diffusiongemma-smoketest/results/code.log new file mode 100644 index 0000000..e782c0c --- /dev/null +++ b/scripts/diffusiongemma-smoketest/results/code.log @@ -0,0 +1,44 @@ +0.00.836.466 W load: control-looking token: 212 '' was not control-type; this is probably a bug in the model. its type will be overridden +0.00.837.141 W load: control-looking token: 50 '<|tool_response>' was not control-type; this is probably a bug in the model. its type will be overridden +0.00.873.644 W load: special_eog_ids contains '<|tool_response>', removing '' token from EOG list +0.04.996.883 I diffusion: -n 160 -> 1 blocks, n_ubatch=2304 n_batch=2304 n_ctx=2304 (canvas_length=256) +0.04.996.886 I diffusion: --fit has no effect here; context is sized from -n and the canvas. Set -ngl / --n-cpu-moe to control device memory. +0.04.997.441 W llama_context: n_ctx_seq (2304) < n_ctx_train (262144) -- the full capacity of the model will not be utilized +0.05.002.413 W sched_reserve: layer 5 is assigned to device CUDA0 but the Flash Attention tensor is assigned to device CPU (usually due to missing support) +0.05.002.416 W sched_reserve: Flash Attention was auto, set to disabled +0.07.343.745 I diffusion_params: steps=128 schedule=0 algorithm=4 temperature=0.800 eps=0.001000 mask_token=4 +0.07.344.014 I diffusion_eb: max_steps=48 t=[0.400,0.800] entropy_bound=0.1000 stability=1 confidence=0.0050 kv_cache=on gpu_sampling=on sample_reduce=on +0.07.345.228 W init: embeddings required but some input tokens were not marked as outputs -> overriding +0.07.822.959 I diffusion step: 0/48 [ ] 0%0.07.925.523 I diffusion step: 1/48 [= ] 2%0.08.028.933 I diffusion step: 2/48 [== ] 4%0.08.132.168 I diffusion step: 3/48 [=== ] 6%0.08.238.755 I diffusion step: 4/48 [==== ] 8%0.08.343.911 I diffusion step: 5/48 [===== ] 10%0.08.450.845 I diffusion step: 6/48 [====== ] 12%0.08.558.796 I diffusion step: 7/48 [======= ] 14%0.08.665.123 I diffusion step: 8/48 [======== ] 16%0.08.771.930 I diffusion step: 9/48 [========= ] 18%0.08.881.776 I diffusion step: 10/48 [========== ] 20%0.08.988.729 I diffusion step: 11/48 [=========== ] 22%0.09.096.269 I diffusion step: 12/48 [============ ] 25% +<|channel>thought +* Task: Write a Python function `is_prime(n)`. + * Requirement: Include a docstring. + * Constraint: Output *only* the code, no explanation. + + * A prime number is a natural number greater than 1 that has no positive divisors other than 1 and itself. + * If $n \le 1$, not prime. + * If $n = 2$, prime. + * If $n$ is even and $>2$, not prime. + * Check divisors from 3 to $\sqrt{n}$. + + ```python + def is_prime(n): + """ + Determines if a number n is prime. + + Args: + n (int): The number to check. + + Returns: + bool: True if n is prime, False otherwise. + """ + if n <= 1: + return False + if n == 2: + return True + if n % 2 == 0: + return False + for i in range(3, int(n**0.5) + 1, +total time: 1751.40ms, time per step: 134.72ms (13 steps over 1 blocks, entropy-bound) +throughput: 146.2 tok/s (256 tok in 1751.40ms), in-step parallel 1900 tok/s (256-tok canvas x 13.0 steps/block) +WALL_SECONDS=9.693752862 RC=0 diff --git a/scripts/diffusiongemma-smoketest/results/format.log b/scripts/diffusiongemma-smoketest/results/format.log new file mode 100644 index 0000000..20fc8cb --- /dev/null +++ b/scripts/diffusiongemma-smoketest/results/format.log @@ -0,0 +1,31 @@ +0.00.841.557 W load: control-looking token: 212 '' was not control-type; this is probably a bug in the model. its type will be overridden +0.00.842.233 W load: control-looking token: 50 '<|tool_response>' was not control-type; this is probably a bug in the model. its type will be overridden +0.00.879.927 W load: special_eog_ids contains '<|tool_response>', removing '' token from EOG list +0.04.814.391 I diffusion: -n 48 -> 1 blocks, n_ubatch=2304 n_batch=2304 n_ctx=2304 (canvas_length=256) +0.04.814.394 I diffusion: --fit has no effect here; context is sized from -n and the canvas. Set -ngl / --n-cpu-moe to control device memory. +0.04.814.907 W llama_context: n_ctx_seq (2304) < n_ctx_train (262144) -- the full capacity of the model will not be utilized +0.04.819.412 W sched_reserve: layer 5 is assigned to device CUDA0 but the Flash Attention tensor is assigned to device CPU (usually due to missing support) +0.04.819.415 W sched_reserve: Flash Attention was auto, set to disabled +0.07.106.365 I diffusion_params: steps=128 schedule=0 algorithm=4 temperature=0.800 eps=0.001000 mask_token=4 +0.07.106.635 I diffusion_eb: max_steps=48 t=[0.400,0.800] entropy_bound=0.1000 stability=1 confidence=0.0050 kv_cache=on gpu_sampling=on sample_reduce=on +0.07.107.839 W init: embeddings required but some input tokens were not marked as outputs -> overriding +0.07.565.506 I diffusion step: 0/48 [ ] 0%0.07.667.965 I diffusion step: 1/48 [= ] 2%0.07.772.514 I diffusion step: 2/48 [== ] 4%0.07.876.431 I diffusion step: 3/48 [=== ] 6%0.07.981.552 I diffusion step: 4/48 [==== ] 8%0.08.087.446 I diffusion step: 5/48 [===== ] 10%0.08.196.277 I diffusion step: 6/48 [====== ] 12%0.08.302.551 I diffusion step: 7/48 [======= ] 14%0.08.407.803 I diffusion step: 8/48 [======== ] 16%0.08.514.697 I diffusion step: 9/48 [========= ] 18%0.08.621.268 I diffusion step: 10/48 [========== ] 20%0.08.728.440 I diffusion step: 11/48 [=========== ] 22%0.08.835.336 I diffusion step: 12/48 [============ ] 25%0.08.942.227 I diffusion step: 13/48 [============= ] 27%0.09.050.835 I diffusion step: 14/48 [============== ] 29%0.09.160.522 I diffusion step: 15/48 [=============== ] 31%0.09.268.593 I diffusion step: 16/48 [================ ] 33%0.09.377.200 I diffusion step: 17/48 [================= ] 35%0.09.485.784 I diffusion step: 18/48 [================== ] 37%0.09.595.474 I diffusion step: 19/48 [=================== ] 39%0.09.704.166 I diffusion step: 20/48 [==================== ] 41% +<|channel>thought +* Topic: Primary colors of light. + * Constraint 1: Exactly five colors. + * Constraint 2: One per line. + * Constraint 3: Nothing else (no intro, no outro). + + * The standard primary colors of light (additive) are Red, Green, and Blue (3 colors). + * Wait, the user is asking for *exactly five*. + * Is there a system with five primary colors of light? + * Standard RGB = Red, Green, Blue (3). + * CMY (subtractive) = Cyan, Magenta, Yellow (3). + * RYB (artistic) = Red, Yellow, Blue (3). + * Some color space models or specific display technologies might use more primaries. + * However, strictly speaking, "primary colors of light" refers to RGB. + * If I must list *five*, I might need to look at a specific model or include the primaries and sub-primary/secondary colors, or perhaps the user is testing the model. + * Actually, usually, when a prompt asks for "exactly X" of +total time: 2596.66ms, time per step: 123.65ms (21 steps over 1 blocks, entropy-bound) +throughput: 98.6 tok/s (256 tok in 2596.66ms), in-step parallel 2070 tok/s (256-tok canvas x 21.0 steps/block) +WALL_SECONDS=10.416765713 RC=0 diff --git a/scripts/diffusiongemma-smoketest/results/format_multiblock.log b/scripts/diffusiongemma-smoketest/results/format_multiblock.log new file mode 100644 index 0000000..c21ab7a --- /dev/null +++ b/scripts/diffusiongemma-smoketest/results/format_multiblock.log @@ -0,0 +1,43 @@ +0.00.778.413 W load: control-looking token: 212 '' was not control-type; this is probably a bug in the model. its type will be overridden +0.00.778.868 W load: control-looking token: 50 '<|tool_response>' was not control-type; this is probably a bug in the model. its type will be overridden +0.00.803.531 W load: special_eog_ids contains '<|tool_response>', removing '' token from EOG list +0.04.672.736 I diffusion: -n 512 -> 2 blocks, n_ubatch=2560 n_batch=2560 n_ctx=2560 (canvas_length=256) +0.04.672.739 I diffusion: --fit has no effect here; context is sized from -n and the canvas. Set -ngl / --n-cpu-moe to control device memory. +0.04.673.245 W llama_context: n_ctx_seq (2560) < n_ctx_train (262144) -- the full capacity of the model will not be utilized +0.04.677.664 W sched_reserve: layer 5 is assigned to device CUDA0 but the Flash Attention tensor is assigned to device CPU (usually due to missing support) +0.04.677.667 W sched_reserve: Flash Attention was auto, set to disabled +0.06.828.575 I diffusion_params: steps=128 schedule=0 algorithm=4 temperature=0.800 eps=0.001000 mask_token=4 +0.06.828.825 I diffusion_eb: max_steps=48 t=[0.400,0.800] entropy_bound=0.1000 stability=1 confidence=0.0050 kv_cache=on gpu_sampling=on sample_reduce=on +0.06.829.987 W init: embeddings required but some input tokens were not marked as outputs -> overriding +0.07.297.536 I diffusion step: 0/48 [ ] 0%0.07.401.754 I diffusion step: 1/48 [= ] 2%0.07.506.767 I diffusion step: 2/48 [== ] 4%0.07.612.593 I diffusion step: 3/48 [=== ] 6%0.07.720.060 I diffusion step: 4/48 [==== ] 8%0.07.827.278 I diffusion step: 5/48 [===== ] 10%0.07.933.928 I diffusion step: 6/48 [====== ] 12%0.08.042.088 I diffusion step: 7/48 [======= ] 14%0.08.150.433 I diffusion step: 8/48 [======== ] 16%0.08.259.575 I diffusion step: 9/48 [========= ] 18%0.08.367.825 I diffusion step: 10/48 [========== ] 20%0.08.475.147 I diffusion step: 11/48 [=========== ] 22%0.08.584.038 I diffusion step: 12/48 [============ ] 25%0.08.693.905 I diffusion step: 13/48 [============= ] 27%0.08.803.010 I diffusion step: 14/48 [============== ] 29%0.08.912.275 I diffusion step: 15/48 [=============== ] 31%0.09.021.574 I diffusion step: 16/48 [================ ] 33%0.09.130.837 I diffusion step: 17/48 [================= ] 35%0.09.240.344 I diffusion step: 18/48 [================== ] 37%0.09.350.482 I diffusion step: 19/48 [=================== ] 39%0.09.458.691 I diffusion step: 20/48 [==================== ] 41%0.09.458.811 W init: embeddings required but some input tokens were not marked as outputs -> overriding +0.09.908.398 I diffusion step: 0/48 [ ] 0%0.10.012.788 I diffusion step: 1/48 [= ] 2%0.10.116.823 I diffusion step: 2/48 [== ] 4%0.10.221.634 I diffusion step: 3/48 [=== ] 6%0.10.327.638 I diffusion step: 4/48 [==== ] 8%0.10.436.834 I diffusion step: 5/48 [===== ] 10%0.10.544.714 I diffusion step: 6/48 [====== ] 12%0.10.652.512 I diffusion step: 7/48 [======= ] 14%0.10.762.174 I diffusion step: 8/48 [======== ] 16%0.10.871.902 I diffusion step: 9/48 [========= ] 18%0.10.984.597 I diffusion step: 10/48 [========== ] 20%0.11.095.808 I diffusion step: 11/48 [=========== ] 22%0.11.207.852 I diffusion step: 12/48 [============ ] 25%0.11.320.651 I diffusion step: 13/48 [============= ] 27%0.11.434.508 I diffusion step: 14/48 [============== ] 29%0.11.547.781 I diffusion step: 15/48 [=============== ] 31%0.11.660.326 I diffusion step: 16/48 [================ ] 33%0.11.773.495 I diffusion step: 17/48 [================= ] 35%0.11.886.310 I diffusion step: 18/48 [================== ] 37%0.12.002.123 I diffusion step: 19/48 [=================== ] 39%0.12.115.707 I diffusion step: 20/48 [==================== ] 41%0.12.229.939 I diffusion step: 21/48 [===================== ] 43%0.12.343.164 I diffusion step: 22/48 [====================== ] 45%0.12.458.359 I diffusion step: 23/48 [======================= ] 47%0.12.572.563 I diffusion step: 24/48 [========================= ] 50%0.12.685.770 I diffusion step: 25/48 [========================== ] 52%0.12.799.935 I diffusion step: 26/48 [=========================== ] 54%0.12.912.457 I diffusion step: 27/48 [============================ ] 56%0.13.028.569 I diffusion step: 28/48 [============================= ] 58%0.13.144.189 I diffusion step: 29/48 [============================== ] 60% +<|channel>thought +* Topic: Primary colors of light. + * Constraint 1: Exactly five colors. + * Constraint 2: One per line. + * Constraint 3: Nothing else (no intro, no outro). + + * The standard primary colors of light (additive) are Red, Green, and Blue (3 colors). + * Wait, the user is asking for *exactly five*. + * Is there a system with five primary colors of light? + * Standard RGB = Red, Green, Blue (3). + * CMY (subtractive) = Cyan, Magenta, Yellow (3). + * RYB (artistic) = Red, Yellow, Blue (3). + * Some color space models or specific display technologies might use more primaries. + * However, strictly speaking, "primary colors of light" refers to RGB. + * If I must list *five*, I might need to look at a specific model or include the primaries and sub-primary/secondary colors, or perhaps the user is testing the model. + * Actually, usually, when a prompt asks for "exactly X" of something that has "Y", it's often a trick or a request for a specific interpretation. + * Is there a 5-primary system? + * Some high-gamut displays use Red, Green, Blue, Cyan, Amber (RGB-CA). + * In some contexts of color theory, people might list more combinations. + * Let's look at the prompt again: "exactly five primary colors of light". + * If I list only 3, I fail the "exactly five" constraint. + * If I list 5, I might be factually incorrect regarding the standard "primary colors of light". + * However, in the context of additive and subtractive primaries or specific spectral models, there might be five. + * Let's check if there's a common "5 primary colors" list. Not really. + * Maybe the user means Red, Green, Blue, Cyan, Magenta? No, those aren't all primaries. + * Wait, perhaps the user is referring to the primary colors of light in a specific context? + * Actually, if I provide 5 colors, I should satisfy the +total time: 6314.54ms, time per step: 123.81ms (51 steps over 2 blocks, entropy-bound) +throughput: 81.1 tok/s (512 tok in 6314.54ms), in-step parallel 2068 tok/s (256-tok canvas x 25.5 steps/block) +WALL_SECONDS=13.802324462 RC=0 diff --git a/scripts/diffusiongemma-smoketest/results/reasoning.log b/scripts/diffusiongemma-smoketest/results/reasoning.log new file mode 100644 index 0000000..8c8a3ec --- /dev/null +++ b/scripts/diffusiongemma-smoketest/results/reasoning.log @@ -0,0 +1,27 @@ +0.00.896.419 W load: control-looking token: 212 '' was not control-type; this is probably a bug in the model. its type will be overridden +0.00.897.085 W load: control-looking token: 50 '<|tool_response>' was not control-type; this is probably a bug in the model. its type will be overridden +0.00.933.523 W load: special_eog_ids contains '<|tool_response>', removing '' token from EOG list +0.04.856.194 I diffusion: -n 128 -> 1 blocks, n_ubatch=2304 n_batch=2304 n_ctx=2304 (canvas_length=256) +0.04.856.197 I diffusion: --fit has no effect here; context is sized from -n and the canvas. Set -ngl / --n-cpu-moe to control device memory. +0.04.864.961 W llama_context: n_ctx_seq (2304) < n_ctx_train (262144) -- the full capacity of the model will not be utilized +0.04.869.425 W sched_reserve: layer 5 is assigned to device CUDA0 but the Flash Attention tensor is assigned to device CPU (usually due to missing support) +0.04.869.427 W sched_reserve: Flash Attention was auto, set to disabled +0.07.226.382 I diffusion_params: steps=128 schedule=0 algorithm=4 temperature=0.800 eps=0.001000 mask_token=4 +0.07.226.626 I diffusion_eb: max_steps=48 t=[0.400,0.800] entropy_bound=0.1000 stability=1 confidence=0.0050 kv_cache=on gpu_sampling=on sample_reduce=on +0.07.228.370 W init: embeddings required but some input tokens were not marked as outputs -> overriding +0.07.721.777 I diffusion step: 0/48 [ ] 0%0.07.822.673 I diffusion step: 1/48 [= ] 2%0.07.924.966 I diffusion step: 2/48 [== ] 4%0.08.030.579 I diffusion step: 3/48 [=== ] 6%0.08.135.224 I diffusion step: 4/48 [==== ] 8%0.08.240.214 I diffusion step: 5/48 [===== ] 10%0.08.346.392 I diffusion step: 6/48 [====== ] 12%0.08.452.589 I diffusion step: 7/48 [======= ] 14%0.08.556.808 I diffusion step: 8/48 [======== ] 16%0.08.663.759 I diffusion step: 9/48 [========= ] 18%0.08.770.846 I diffusion step: 10/48 [========== ] 20%0.08.876.181 I diffusion step: 11/48 [=========== ] 22%0.08.984.389 I diffusion step: 12/48 [============ ] 25%0.09.090.492 I diffusion step: 13/48 [============= ] 27%0.09.198.211 I diffusion step: 14/48 [============== ] 29%0.09.304.460 I diffusion step: 15/48 [=============== ] 31%0.09.410.010 I diffusion step: 16/48 [================ ] 33%0.09.516.289 I diffusion step: 17/48 [================= ] 35%0.09.623.885 I diffusion step: 18/48 [================== ] 37% +<|channel>thought +* Topic: Why diffusion language models are faster than autoregressive ones. + * Constraint: Exactly three sentences. + + * *Autoregressive (AR) models:* Generate one token at a time (sequentially). To generate a sequence of length $N$, they need $N$ forward passes through the neural network. This is $O(N)$ complexity. + * *Diffusion models:* Generate the entire sequence in parallel (or through a few iterative denoising steps). While they require multiple steps, the number of steps is often independent of the sequence length or significantly fewer than the length for long sequences. + + * *Sentence 1:* Autoregressive models must generate tokens one by one, meaning the number of required forward passes scales linearly with the length of the output sequence. + * *Sentence 2:* In contrast, diffusion models can refine the entire sequence simultaneously through a fixed or limited number of denoising steps. + * *Sentence 3:* Because the number of iterations in diffusion models is often constant or independent of the total sequence length, they can achieve faster inference speeds for long-form text. + + * Sentence 1: Autoregressive models generate tokens sequentially, +total time: 2396.01ms, time per step: 126.11ms (19 steps over 1 blocks, entropy-bound) +throughput: 106.8 tok/s (256 tok in 2396.01ms), in-step parallel 2030 tok/s (256-tok canvas x 19.0 steps/block) +WALL_SECONDS=10.156527687 RC=0 diff --git a/scripts/diffusiongemma-smoketest/run.sh b/scripts/diffusiongemma-smoketest/run.sh new file mode 100755 index 0000000..208dbfb --- /dev/null +++ b/scripts/diffusiongemma-smoketest/run.sh @@ -0,0 +1,46 @@ +#!/usr/bin/env bash +# DiffusionGemma smoke test — steel141 3090 Ti +# +# DiffusionGemma (google/diffusiongemma-26B-A4B-it, released 2026-06-10) is a +# text-DIFFUSION MoE model. It does NOT run in Ollama or stock llama.cpp: +# the standard llama-cli/llama-server reject arch 'diffusion-gemma'. It needs +# the dedicated `llama-diffusion-cli` binary from ggml-org/llama.cpp PR #24423 +# (danielhanchen / Unsloth), which denoises 256-token canvas blocks in parallel. +# +# This harness runs a few non-interactive prompts and captures wall-clock + the +# diffusion step/block telemetry the CLI prints. +set -euo pipefail + +CLI="${CLI:-/mnt/ai_data/diffusiongemma/llama.cpp/build/bin/llama-diffusion-cli}" +MODEL="${MODEL:-/mnt/ai_data/diffusiongemma/gguf/diffusiongemma-26B-A4B-it-Q4_K_M.gguf}" +NGL="${NGL:-99}" # offload all layers to GPU +NPRED="${NPRED:-256}" # one full canvas +OUTDIR="${OUTDIR:-$(dirname "$0")/results}" +mkdir -p "$OUTDIR" + +[ -x "$CLI" ] || { echo "missing CLI: $CLI" >&2; exit 1; } +[ -f "$MODEL" ] || { echo "missing model: $MODEL" >&2; exit 1; } + +run() { + local name="$1"; local prompt="$2"; shift 2 + local log="$OUTDIR/${name}.log" + echo "=== $name ===" | tee "$log" + echo "prompt: $prompt" | tee -a "$log" + local t0 t1 + t0=$(date +%s.%N) + "$CLI" -m "$MODEL" -ngl "$NGL" -n "$NPRED" --diffusion-eb auto -p "$prompt" "$@" 2>&1 | tee -a "$log" + t1=$(date +%s.%N) + echo "WALL_SECONDS=$(echo "$t1 - $t0" | bc)" | tee -a "$log" + echo | tee -a "$log" +} + +# 1) Plain reasoning — sanity + coherence +run reasoning "Explain in three sentences why diffusion language models can be faster than autoregressive ones." + +# 2) Code — structured output the diffusion canvas has to fill coherently +run code "Write a Python function is_prime(n) with a docstring. Output only the code." + +# 3) Instruction following with a hard format constraint +run format "List exactly five primary colors of light, one per line, no extra text." + +echo "All runs complete. Logs in $OUTDIR"