From a662df684eb6d942d96d390cce909b6b5f30ecb9 Mon Sep 17 00:00:00 2001 From: Mortdecai Date: Wed, 17 Jun 2026 22:11:52 -0400 Subject: [PATCH] =?UTF-8?q?docs:=20session=20handoff=20=E2=80=94=20Diffusi?= =?UTF-8?q?onGemma=20research=20+=20smoke=20test=20on=203090=20Ti?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Opus 4.8 (1M context) --- ...21048-diffusiongemma-research-smoketest.md | 140 ++++++++++++++++++ 1 file changed, 140 insertions(+) create mode 100644 .claude/handoffs/2026-06-17-221048-diffusiongemma-research-smoketest.md diff --git a/.claude/handoffs/2026-06-17-221048-diffusiongemma-research-smoketest.md b/.claude/handoffs/2026-06-17-221048-diffusiongemma-research-smoketest.md new file mode 100644 index 0000000..a2e743a --- /dev/null +++ b/.claude/handoffs/2026-06-17-221048-diffusiongemma-research-smoketest.md @@ -0,0 +1,140 @@ +# Handoff: DiffusionGemma Research + Smoke Test (built llama-diffusion-cli on 3090 Ti) + +## Session Metadata +- Created: 2026-06-17 22:10:48 +- Project: /home/claude/bin/gemma4-research +- Branch: master +- Session duration: ~1.5 hours + +### Recent Commits (for context) + - 6bce2d4 docs: DiffusionGemma research + first-hand smoke test on 3090 Ti + - 043a9ae docs: session handoff — gemma4:12b research + smoke test + steel141 Ollama 0.30.8 upgrade + - 2336fb4 docs: add gemma4:12b variant + smoke-test results + - f11aa31 docs: session handoff — Gemma 4 tool-calling May-2026 update + bot audit + - 438a96f docs: May 2026 tool-calling update + agentic retrieval landscape + +## Handoff Chain + +- **Continues from**: [2026-06-16-230331-gemma4-12b-research-smoketest.md](./2026-06-16-230331-gemma4-12b-research-smoketest.md) + - Previous title: gemma4:12b Research + Smoke Test (and steel141 Ollama 0.20.4→0.30.8 upgrade) +- **Supersedes**: None + +## Current State Summary + +Researched and smoke-tested **DiffusionGemma** (`google/diffusiongemma-26B-A4B-it`, released 2026-06-10) — Google's first open-weight **text-diffusion** LLM. The headline finding: it does **NOT** run in the homelab Ollama stack (`unknown model architecture: 'diffusion-gemma'`) and won't until the diffusion arch merges to llama.cpp mainline. Seth chose the "build the llama.cpp fork on the 3090 Ti" path. I installed the CUDA toolkit on steel141 (was missing), built `llama-diffusion-cli` from ggml-org/llama.cpp **PR #24423**, pulled the Q4_K_M GGUF (16.8 GB), and ran a 4-prompt smoke test on the 3090 Ti. Findings committed (6bce2d4) and pushed. Work is **complete** — only optional follow-ups remain. + +## Codebase Understanding + +### Architecture Overview + +Documentation/research corpus repo (no app code). Findings live in `CORPUS_*.md` + `docs/reference/*.md`, indexed by `README.md`. Smoke tests normally hit local Ollama, but DiffusionGemma needed a custom-built `llama-diffusion-cli` binary instead. + +### Critical Files + +| File | Purpose | Relevance | +|------|---------|-----------| +| `docs/reference/diffusiongemma-smoketest-2026-06-17.md` | **Full writeup** — specs, build recipe, throughput, gotchas | Created this session | +| `scripts/diffusiongemma-smoketest/run.sh` + `results/*.log` | Harness + raw smoke-test logs | Created this session | +| `CORPUS_ollama_variants.md` | Added "DiffusionGemma is NOT an Ollama variant" callout | Edited this session | +| `README.md` | Added reference-doc index line | Edited this session | + +### Key Patterns Discovered + +- **DiffusionGemma generation**: denoises a 256-token canvas in parallel via an entropy-bound decoder (13–25 steps/block, ~124 ms/step), not autoregressive. Two throughput numbers: "in-step parallel" (~2030 tok/s, canvas-parallel) vs **effective** (~106 tok/s end-to-end). +- **`<|channel>thought` CoT channel** is denoised *inside* the canvas and eats it — strict short-format prompts truncate before the answer unless you budget `--diffusion-blocks` ≥ 4. + +## Work Completed + +### Tasks Finished + +- [x] Researched DiffusionGemma (specs, modalities, run paths) — released 2026-06-10, Apache-2.0, 25.2B/3.8B-active MoE + diffusion head +- [x] Confirmed it does NOT run in Ollama/llama-cli (arch `diffusion-gemma`); needs `llama-diffusion-cli` from PR #24423 +- [x] Installed `nvidia-cuda-toolkit` 12.4 on steel141 (nvcc was absent) +- [x] Built `llama-diffusion-cli` (CUDA sm_86) at `/mnt/ai_data/diffusiongemma/llama.cpp/build/bin/` +- [x] Downloaded Q4_K_M GGUF (16.8 GB) to `/mnt/ai_data/diffusiongemma/gguf/` +- [x] Smoke test on 3090 Ti: reasoning ✅ coherent, code ✅ correct `is_prime`, format ⚠️ thought-channel spiral +- [x] Wrote reference doc, updated corpus + README, committed (6bce2d4) + pushed +- [x] Saved 2 memories (DiffusionGemma-not-in-Ollama, steel141 GPU/CUDA ordering) + +### Files Modified + +| File | Changes | Rationale | +|------|---------|-----------| +| `CORPUS_ollama_variants.md` | "not an Ollama variant" callout block | Stop anyone trying `ollama pull diffusiongemma` | +| `README.md` | Reference-doc index row | Discoverability | +| `docs/reference/diffusiongemma-smoketest-2026-06-17.md` | New | Full findings + build recipe | +| `scripts/diffusiongemma-smoketest/` | New harness + result logs | Reproducibility | + +### Decisions Made + +| Decision | Options Considered | Rationale | +|----------|-------------------|-----------| +| llama.cpp fork on 3090 Ti | (a) fork+GGUF on 3090 Ti, (b) Transformers 4-bit, (c) cloud H100 vLLM | Seth chose (a) — local, free, closest to the GGUF path he'd deploy on once it merges | +| Install CUDA toolkit via apt | manual CUDA tarball | apt 12.4 matches the 550 driver's CUDA 12.4 exactly; reversible | +| Stop ollama per run window | `ollama stop ` only | stop-model loses the race to live traffic that re-loads gemma4:26b; full service stop is deterministic, bounded ~3 min | + +## Pending Work + +### Immediate Next Steps + +1. **(Optional) Re-run with thinking suppressed / more blocks** to get clean short-format outputs — the `<|channel>thought` channel needs steering. Try a system prompt that forbids the thought channel, or `--diffusion-blocks 6+`. +2. **(Optional) Vision smoke test** — DiffusionGemma takes image+video input; only text was tested. Would need the mmproj path through `llama-diffusion-cli` / `diffusion-gemma-server`. +3. **(Optional) Benchmark vs autoregressive gemma4:26b** on matched prompts to quantify the diffusion speed tradeoff at longer canvases (where diffusion should pull ahead). + +### Blockers/Open Questions + +- [ ] DiffusionGemma stays an experimental artifact until llama.cpp mainline merges the diffusion arch (then Ollama). No deployment path for Simon/AI_Visualizer/mort-bot yet. + +### Deferred Items + +- Did not test the `diffusion-gemma-server` (HTTP) build target — only the CLI. The server exists in PR #24423 (`examples/diffusion-gemma-server/`) if an API smoke test is wanted later. +- Did not delete the 16.8 GB GGUF or the llama.cpp build — left at `/mnt/ai_data/diffusiongemma/` for reproducibility (325 GB free there). + +## Context for Resuming Agent + +### Important Context + +The model is **not** Ollama-pullable — that's the whole point of the writeup. Anyone wanting to run it again uses `/mnt/ai_data/diffusiongemma/llama.cpp/build/bin/llama-diffusion-cli` with the Q4_K_M GGUF in the sibling `gguf/` dir. **Select the 3090 Ti by UUID** (`CUDA_VISIBLE_DEVICES=GPU-61fed72d-0698-c3b1-a789-6f93a1ed52d9`) — nvidia-smi index ≠ CUDA index on steel141 (this burned two runs). And **stop ollama first** to free VRAM, restart after. + +### Assumptions Made + +- CUDA toolkit 12.4 (apt) is compatible with driver 550.163.01 / CUDA 12.4 — verified, build + inference both worked. +- The apt-installed toolkit and the 16.8 GB GGUF are fine to leave on steel141 (disk has headroom). If Seth wants them gone: `sudo apt remove nvidia-cuda-toolkit` and `rm -rf /mnt/ai_data/diffusiongemma`. + +### Potential Gotchas + +- **nvidia-smi GPU index ≠ CUDA device index on steel141** (3090 Ti is nvidia-smi GPU 1 but CUDA device 0). Select by UUID. +- `/usr/bin/time` is not installed on steel141 — use shell `date +%s.%N` for timing. +- `/mnt/ai_data` is owned by `ollama:ollama` — needed `sudo mkdir + chown claude` to write there. +- `gh` is not installed — fetch PRs with `git fetch origin pull/N/head:branch`. +- The `<|channel>thought` CoT eats the 256-token canvas; budget blocks or steer it. + +## Environment State + +### Tools/Services Used + +- steel141 Ollama daemon (v0.30.8) — **stopped and restarted** several times during the smoke test; currently **active** and healthy (verified `curl /api/tags`). +- 3090 Ti (nvidia-smi GPU 1 / CUDA device 0) for inference; restored to idle. +- New: `nvidia-cuda-toolkit` 12.4.131 (`/usr/bin/nvcc`), installed this session. +- Built binary: `/mnt/ai_data/diffusiongemma/llama.cpp/build/bin/llama-diffusion-cli`. + +### Active Processes + +- `ollama serve` under systemd (`systemctl is-active ollama` = active). No background jobs left from this session. + +### Environment Variables + +- `CUDA_VISIBLE_DEVICES` (set to the 3090 Ti UUID for runs — not persisted) +- Ollama's `OLLAMA_HOST` / `OLLAMA_KEEP_ALIVE` (systemd override, unchanged) + +## Related Resources + +- `docs/reference/diffusiongemma-smoketest-2026-06-17.md` — full writeup + build recipe +- `scripts/diffusiongemma-smoketest/run.sh` + `results/` — harness + raw logs +- `/mnt/ai_data/diffusiongemma/` — GGUF + llama.cpp build (local, not in git) +- Memories: `project_diffusiongemma_not_in_ollama`, `project_steel141_gpu_cuda_ordering` +- Web: ai.google.dev/gemma/docs/diffusiongemma · huggingface.co/google/diffusiongemma-26B-A4B-it · ggml-org/llama.cpp PR #24423 · ollama/ollama#16664 + +--- + +**Security Reminder**: Before finalizing, run `validate_handoff.py` to check for accidental secret exposure.