gemma4-research/.claude/handoffs/2026-06-17-221048-diffusiongemma-research-smoketest.md

Handoff: DiffusionGemma Research + Smoke Test (built llama-diffusion-cli on 3090 Ti)

Session Metadata

• Created: 2026-06-17 22:10:48
• Project: /home/claude/bin/gemma4-research
• Branch: master
• Session duration: ~1.5 hours

Recent Commits (for context)

• 6bce2d4 docs: DiffusionGemma research + first-hand smoke test on 3090 Ti
• 043a9ae docs: session handoff — gemma4:12b research + smoke test + steel141 Ollama 0.30.8 upgrade
• 2336fb4 docs: add gemma4:12b variant + smoke-test results
• f11aa31 docs: session handoff — Gemma 4 tool-calling May-2026 update + bot audit
• 438a96f docs: May 2026 tool-calling update + agentic retrieval landscape

Handoff Chain

• Continues from: 2026-06-16-230331-gemma4-12b-research-smoketest.md
  • Previous title: gemma4:12b Research + Smoke Test (and steel141 Ollama 0.20.4→0.30.8 upgrade)
• Supersedes: None

Current State Summary

Researched and smoke-tested DiffusionGemma (google/diffusiongemma-26B-A4B-it, released 2026-06-10) — Google's first open-weight text-diffusion LLM. The headline finding: it does NOT run in the homelab Ollama stack (unknown model architecture: 'diffusion-gemma') and won't until the diffusion arch merges to llama.cpp mainline. Seth chose the "build the llama.cpp fork on the 3090 Ti" path. I installed the CUDA toolkit on steel141 (was missing), built llama-diffusion-cli from ggml-org/llama.cpp PR #24423, pulled the Q4_K_M GGUF (16.8 GB), and ran a 4-prompt smoke test on the 3090 Ti. Findings committed (6bce2d4) and pushed. Work is complete — only optional follow-ups remain.

Codebase Understanding

Architecture Overview

Documentation/research corpus repo (no app code). Findings live in CORPUS_*.md + docs/reference/*.md, indexed by README.md. Smoke tests normally hit local Ollama, but DiffusionGemma needed a custom-built llama-diffusion-cli binary instead.

Critical Files

File	Purpose	Relevance
docs/reference/diffusiongemma-smoketest-2026-06-17.md	Full writeup — specs, build recipe, throughput, gotchas	Created this session
scripts/diffusiongemma-smoketest/run.sh + results/*.log	Harness + raw smoke-test logs	Created this session
CORPUS_ollama_variants.md	Added "DiffusionGemma is NOT an Ollama variant" callout	Edited this session
README.md	Added reference-doc index line	Edited this session

Key Patterns Discovered

• DiffusionGemma generation: denoises a 256-token canvas in parallel via an entropy-bound decoder (13–25 steps/block, ~124 ms/step), not autoregressive. Two throughput numbers: "in-step parallel" (~2030 tok/s, canvas-parallel) vs effective (~106 tok/s end-to-end).
• <|channel>thought CoT channel is denoised inside the canvas and eats it — strict short-format prompts truncate before the answer unless you budget --diffusion-blocks ≥ 4.

Work Completed

Tasks Finished

• Researched DiffusionGemma (specs, modalities, run paths) — released 2026-06-10, Apache-2.0, 25.2B/3.8B-active MoE + diffusion head
• Confirmed it does NOT run in Ollama/llama-cli (arch diffusion-gemma); needs llama-diffusion-cli from PR #24423
• Installed nvidia-cuda-toolkit 12.4 on steel141 (nvcc was absent)
• Built llama-diffusion-cli (CUDA sm_86) at /mnt/ai_data/diffusiongemma/llama.cpp/build/bin/
• Downloaded Q4_K_M GGUF (16.8 GB) to /mnt/ai_data/diffusiongemma/gguf/
• Smoke test on 3090 Ti: reasoning ✅ coherent, code ✅ correct is_prime, format ⚠️ thought-channel spiral
• Wrote reference doc, updated corpus + README, committed (6bce2d4) + pushed
• Saved 2 memories (DiffusionGemma-not-in-Ollama, steel141 GPU/CUDA ordering)

Files Modified

File	Changes	Rationale
CORPUS_ollama_variants.md	"not an Ollama variant" callout block	Stop anyone trying ollama pull diffusiongemma
README.md	Reference-doc index row	Discoverability
docs/reference/diffusiongemma-smoketest-2026-06-17.md	New	Full findings + build recipe
scripts/diffusiongemma-smoketest/	New harness + result logs	Reproducibility

Decisions Made

Decision	Options Considered	Rationale
llama.cpp fork on 3090 Ti	(a) fork+GGUF on 3090 Ti, (b) Transformers 4-bit, (c) cloud H100 vLLM	Seth chose (a) — local, free, closest to the GGUF path he'd deploy on once it merges
Install CUDA toolkit via apt	manual CUDA tarball	apt 12.4 matches the 550 driver's CUDA 12.4 exactly; reversible
Stop ollama per run window	ollama stop <model> only	stop-model loses the race to live traffic that re-loads gemma4:26b; full service stop is deterministic, bounded ~3 min

Pending Work

Immediate Next Steps

1. (Optional) Re-run with thinking suppressed / more blocks to get clean short-format outputs — the <|channel>thought channel needs steering. Try a system prompt that forbids the thought channel, or --diffusion-blocks 6+.
2. (Optional) Vision smoke test — DiffusionGemma takes image+video input; only text was tested. Would need the mmproj path through llama-diffusion-cli / diffusion-gemma-server.
3. (Optional) Benchmark vs autoregressive gemma4:26b on matched prompts to quantify the diffusion speed tradeoff at longer canvases (where diffusion should pull ahead).

Blockers/Open Questions

• DiffusionGemma stays an experimental artifact until llama.cpp mainline merges the diffusion arch (then Ollama). No deployment path for Simon/AI_Visualizer/mort-bot yet.

Deferred Items

• Did not test the diffusion-gemma-server (HTTP) build target — only the CLI. The server exists in PR #24423 (examples/diffusion-gemma-server/) if an API smoke test is wanted later.
• Did not delete the 16.8 GB GGUF or the llama.cpp build — left at /mnt/ai_data/diffusiongemma/ for reproducibility (325 GB free there).

Context for Resuming Agent

Important Context

The model is not Ollama-pullable — that's the whole point of the writeup. Anyone wanting to run it again uses /mnt/ai_data/diffusiongemma/llama.cpp/build/bin/llama-diffusion-cli with the Q4_K_M GGUF in the sibling gguf/ dir. Select the 3090 Ti by UUID (CUDA_VISIBLE_DEVICES=GPU-61fed72d-0698-c3b1-a789-6f93a1ed52d9) — nvidia-smi index ≠ CUDA index on steel141 (this burned two runs). And stop ollama first to free VRAM, restart after.

Assumptions Made

• CUDA toolkit 12.4 (apt) is compatible with driver 550.163.01 / CUDA 12.4 — verified, build + inference both worked.
• The apt-installed toolkit and the 16.8 GB GGUF are fine to leave on steel141 (disk has headroom). If Seth wants them gone: sudo apt remove nvidia-cuda-toolkit and rm -rf /mnt/ai_data/diffusiongemma.

Potential Gotchas

• nvidia-smi GPU index ≠ CUDA device index on steel141 (3090 Ti is nvidia-smi GPU 1 but CUDA device 0). Select by UUID.
• /usr/bin/time is not installed on steel141 — use shell date +%s.%N for timing.
• /mnt/ai_data is owned by ollama:ollama — needed sudo mkdir + chown claude to write there.
• gh is not installed — fetch PRs with git fetch origin pull/N/head:branch.
• The <|channel>thought CoT eats the 256-token canvas; budget blocks or steer it.

Environment State

Tools/Services Used

• steel141 Ollama daemon (v0.30.8) — stopped and restarted several times during the smoke test; currently active and healthy (verified curl /api/tags).
• 3090 Ti (nvidia-smi GPU 1 / CUDA device 0) for inference; restored to idle.
• New: nvidia-cuda-toolkit 12.4.131 (/usr/bin/nvcc), installed this session.
• Built binary: /mnt/ai_data/diffusiongemma/llama.cpp/build/bin/llama-diffusion-cli.

Active Processes

• ollama serve under systemd (systemctl is-active ollama = active). No background jobs left from this session.

Environment Variables

• CUDA_VISIBLE_DEVICES (set to the 3090 Ti UUID for runs — not persisted)
• Ollama's OLLAMA_HOST / OLLAMA_KEEP_ALIVE (systemd override, unchanged)

Related Resources

• docs/reference/diffusiongemma-smoketest-2026-06-17.md — full writeup + build recipe
• scripts/diffusiongemma-smoketest/run.sh + results/ — harness + raw logs
• /mnt/ai_data/diffusiongemma/ — GGUF + llama.cpp build (local, not in git)
• Memories: project_diffusiongemma_not_in_ollama, project_steel141_gpu_cuda_ordering
• Web: ai.google.dev/gemma/docs/diffusiongemma · huggingface.co/google/diffusiongemma-26B-A4B-it · ggml-org/llama.cpp PR #24423 · ollama/ollama#16664

---

Security Reminder: Before finalizing, run validate_handoff.py to check for accidental secret exposure.