Mortdecai
6bce2d4c3b
google/diffusiongemma-26B-A4B-it (released 2026-06-10) — Google's first open-weight text-diffusion LLM. Does NOT run in Ollama (unknown arch 'diffusion-gemma'); built llama-diffusion-cli from ggml-org/llama.cpp PR #24423 and smoke-tested Q4_K_M on steel141's 3090 Ti. - New reference doc with specs, build recipe, throughput, and gotchas - CORPUS_ollama_variants.md: "not an Ollama variant" callout - README index line for the reference doc - scripts/diffusiongemma-smoketest/ harness + raw result logs Findings: ~106 tok/s effective / ~2030 tok/s in-step-parallel; correct code + coherent reasoning; <|channel>thought CoT eats the 256-tok canvas so strict short formats need block budgeting. nvidia-smi index != CUDA index on steel141 (select 3090 Ti by UUID). Experimental research artifact, not homelab-deployable until diffusion arch merges to llama.cpp mainline. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
28 lines
4.5 KiB
Plaintext
28 lines
4.5 KiB
Plaintext
0.00.896.419 W load: control-looking token: 212 '</s>' was not control-type; this is probably a bug in the model. its type will be overridden
|
|
0.00.897.085 W load: control-looking token: 50 '<|tool_response>' was not control-type; this is probably a bug in the model. its type will be overridden
|
|
0.00.933.523 W load: special_eog_ids contains '<|tool_response>', removing '</s>' token from EOG list
|
|
0.04.856.194 I diffusion: -n 128 -> 1 blocks, n_ubatch=2304 n_batch=2304 n_ctx=2304 (canvas_length=256)
|
|
0.04.856.197 I diffusion: --fit has no effect here; context is sized from -n and the canvas. Set -ngl / --n-cpu-moe to control device memory.
|
|
0.04.864.961 W llama_context: n_ctx_seq (2304) < n_ctx_train (262144) -- the full capacity of the model will not be utilized
|
|
0.04.869.425 W sched_reserve: layer 5 is assigned to device CUDA0 but the Flash Attention tensor is assigned to device CPU (usually due to missing support)
|
|
0.04.869.427 W sched_reserve: Flash Attention was auto, set to disabled
|
|
0.07.226.382 I diffusion_params: steps=128 schedule=0 algorithm=4 temperature=0.800 eps=0.001000 mask_token=4
|
|
0.07.226.626 I diffusion_eb: max_steps=48 t=[0.400,0.800] entropy_bound=0.1000 stability=1 confidence=0.0050 kv_cache=on gpu_sampling=on sample_reduce=on
|
|
0.07.228.370 W init: embeddings required but some input tokens were not marked as outputs -> overriding
|
|
0.07.721.777 I
|
|
diffusion step: 0/48 [ ] 0%0.07.822.673 I
|
|
diffusion step: 1/48 [= ] 2%0.07.924.966 I
|
|
diffusion step: 2/48 [== ] 4%0.08.030.579 I
|
|
diffusion step: 3/48 [=== ] 6%0.08.135.224 I
|
|
diffusion step: 4/48 [==== ] 8%0.08.240.214 I
|
|
diffusion step: 5/48 [===== ] 10%0.08.346.392 I
|
|
diffusion step: 6/48 [====== ] 12%0.08.452.589 I
|
|
diffusion step: 7/48 [======= ] 14%0.08.556.808 I
|
|
diffusion step: 8/48 [======== ] 16%0.08.663.759 I
|
|
diffusion step: 9/48 [========= ] 18%0.08.770.846 I
|
|
diffusion step: 10/48 [========== ] 20%0.08.876.181 I
|
|
diffusion step: 11/48 [=========== ] 22%0.08.984.389 I
|
|
diffusion step: 12/48 [============ ] 25%0.09.090.492 I
|
|
diffusion step: 13/48 [============= ] 27%0.09.198.211 I
|
|
diffusion step: 14/48 [============== ] 29%0.09.304.460 I
|
|
diffusion step: 15/48 [=============== ] 31%0.09.410.010 I
|
|
diffusion step: 16/48 [================ ] 33%0.09.516.289 I
|
|
diffusion step: 17/48 [================= ] 35%0.09.623.885 I
|
|
diffusion step: 18/48 [================== ] 37%
|
|
<|channel>thought
|
|
* Topic: Why diffusion language models are faster than autoregressive ones.
|
|
* Constraint: Exactly three sentences.
|
|
|
|
* *Autoregressive (AR) models:* Generate one token at a time (sequentially). To generate a sequence of length $N$, they need $N$ forward passes through the neural network. This is $O(N)$ complexity.
|
|
* *Diffusion models:* Generate the entire sequence in parallel (or through a few iterative denoising steps). While they require multiple steps, the number of steps is often independent of the sequence length or significantly fewer than the length for long sequences.
|
|
|
|
* *Sentence 1:* Autoregressive models must generate tokens one by one, meaning the number of required forward passes scales linearly with the length of the output sequence.
|
|
* *Sentence 2:* In contrast, diffusion models can refine the entire sequence simultaneously through a fixed or limited number of denoising steps.
|
|
* *Sentence 3:* Because the number of iterations in diffusion models is often constant or independent of the total sequence length, they can achieve faster inference speeds for long-form text.
|
|
|
|
* Sentence 1: Autoregressive models generate tokens sequentially,
|
|
total time: 2396.01ms, time per step: 126.11ms (19 steps over 1 blocks, entropy-bound)
|
|
throughput: 106.8 tok/s (256 tok in 2396.01ms), in-step parallel 2030 tok/s (256-tok canvas x 19.0 steps/block)
|
|
WALL_SECONDS=10.156527687 RC=0
|