docs: session handoff — DiffusionGemma research + smoke test on 3090 Ti
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Mortdecai
@@ -0,0 +1,140 @@
|
||||
# Handoff: DiffusionGemma Research + Smoke Test (built llama-diffusion-cli on 3090 Ti)
|
||||
|
||||
## Session Metadata
|
||||
- Created: 2026-06-17 22:10:48
|
||||
- Project: /home/claude/bin/gemma4-research
|
||||
- Branch: master
|
||||
- Session duration: ~1.5 hours
|
||||
|
||||
### Recent Commits (for context)
|
||||
- 6bce2d4 docs: DiffusionGemma research + first-hand smoke test on 3090 Ti
|
||||
- 043a9ae docs: session handoff — gemma4:12b research + smoke test + steel141 Ollama 0.30.8 upgrade
|
||||
- 2336fb4 docs: add gemma4:12b variant + smoke-test results
|
||||
- f11aa31 docs: session handoff — Gemma 4 tool-calling May-2026 update + bot audit
|
||||
- 438a96f docs: May 2026 tool-calling update + agentic retrieval landscape
|
||||
|
||||
## Handoff Chain
|
||||
|
||||
- **Continues from**: [2026-06-16-230331-gemma4-12b-research-smoketest.md](./2026-06-16-230331-gemma4-12b-research-smoketest.md)
|
||||
- Previous title: gemma4:12b Research + Smoke Test (and steel141 Ollama 0.20.4→0.30.8 upgrade)
|
||||
- **Supersedes**: None
|
||||
|
||||
## Current State Summary
|
||||
|
||||
Researched and smoke-tested **DiffusionGemma** (`google/diffusiongemma-26B-A4B-it`, released 2026-06-10) — Google's first open-weight **text-diffusion** LLM. The headline finding: it does **NOT** run in the homelab Ollama stack (`unknown model architecture: 'diffusion-gemma'`) and won't until the diffusion arch merges to llama.cpp mainline. Seth chose the "build the llama.cpp fork on the 3090 Ti" path. I installed the CUDA toolkit on steel141 (was missing), built `llama-diffusion-cli` from ggml-org/llama.cpp **PR #24423**, pulled the Q4_K_M GGUF (16.8 GB), and ran a 4-prompt smoke test on the 3090 Ti. Findings committed (6bce2d4) and pushed. Work is **complete** — only optional follow-ups remain.
|
||||
|
||||
## Codebase Understanding
|
||||
|
||||
### Architecture Overview
|
||||
|
||||
Documentation/research corpus repo (no app code). Findings live in `CORPUS_*.md` + `docs/reference/*.md`, indexed by `README.md`. Smoke tests normally hit local Ollama, but DiffusionGemma needed a custom-built `llama-diffusion-cli` binary instead.
|
||||
|
||||
### Critical Files
|
||||
|
||||
| File | Purpose | Relevance |
|
||||
|------|---------|-----------|
|
||||
| `docs/reference/diffusiongemma-smoketest-2026-06-17.md` | **Full writeup** — specs, build recipe, throughput, gotchas | Created this session |
|
||||
| `scripts/diffusiongemma-smoketest/run.sh` + `results/*.log` | Harness + raw smoke-test logs | Created this session |
|
||||
| `CORPUS_ollama_variants.md` | Added "DiffusionGemma is NOT an Ollama variant" callout | Edited this session |
|
||||
| `README.md` | Added reference-doc index line | Edited this session |
|
||||
|
||||
### Key Patterns Discovered
|
||||
|
||||
- **DiffusionGemma generation**: denoises a 256-token canvas in parallel via an entropy-bound decoder (13–25 steps/block, ~124 ms/step), not autoregressive. Two throughput numbers: "in-step parallel" (~2030 tok/s, canvas-parallel) vs **effective** (~106 tok/s end-to-end).
|
||||
- **`<|channel>thought` CoT channel** is denoised *inside* the canvas and eats it — strict short-format prompts truncate before the answer unless you budget `--diffusion-blocks` ≥ 4.
|
||||
|
||||
## Work Completed
|
||||
|
||||
### Tasks Finished
|
||||
|
||||
- [x] Researched DiffusionGemma (specs, modalities, run paths) — released 2026-06-10, Apache-2.0, 25.2B/3.8B-active MoE + diffusion head
|
||||
- [x] Confirmed it does NOT run in Ollama/llama-cli (arch `diffusion-gemma`); needs `llama-diffusion-cli` from PR #24423
|
||||
- [x] Installed `nvidia-cuda-toolkit` 12.4 on steel141 (nvcc was absent)
|
||||
- [x] Built `llama-diffusion-cli` (CUDA sm_86) at `/mnt/ai_data/diffusiongemma/llama.cpp/build/bin/`
|
||||
- [x] Downloaded Q4_K_M GGUF (16.8 GB) to `/mnt/ai_data/diffusiongemma/gguf/`
|
||||
- [x] Smoke test on 3090 Ti: reasoning ✅ coherent, code ✅ correct `is_prime`, format ⚠️ thought-channel spiral
|
||||
- [x] Wrote reference doc, updated corpus + README, committed (6bce2d4) + pushed
|
||||
- [x] Saved 2 memories (DiffusionGemma-not-in-Ollama, steel141 GPU/CUDA ordering)
|
||||
|
||||
### Files Modified
|
||||
|
||||
| File | Changes | Rationale |
|
||||
|------|---------|-----------|
|
||||
| `CORPUS_ollama_variants.md` | "not an Ollama variant" callout block | Stop anyone trying `ollama pull diffusiongemma` |
|
||||
| `README.md` | Reference-doc index row | Discoverability |
|
||||
| `docs/reference/diffusiongemma-smoketest-2026-06-17.md` | New | Full findings + build recipe |
|
||||
| `scripts/diffusiongemma-smoketest/` | New harness + result logs | Reproducibility |
|
||||
|
||||
### Decisions Made
|
||||
|
||||
| Decision | Options Considered | Rationale |
|
||||
|----------|-------------------|-----------|
|
||||
| llama.cpp fork on 3090 Ti | (a) fork+GGUF on 3090 Ti, (b) Transformers 4-bit, (c) cloud H100 vLLM | Seth chose (a) — local, free, closest to the GGUF path he'd deploy on once it merges |
|
||||
| Install CUDA toolkit via apt | manual CUDA tarball | apt 12.4 matches the 550 driver's CUDA 12.4 exactly; reversible |
|
||||
| Stop ollama per run window | `ollama stop <model>` only | stop-model loses the race to live traffic that re-loads gemma4:26b; full service stop is deterministic, bounded ~3 min |
|
||||
|
||||
## Pending Work
|
||||
|
||||
### Immediate Next Steps
|
||||
|
||||
1. **(Optional) Re-run with thinking suppressed / more blocks** to get clean short-format outputs — the `<|channel>thought` channel needs steering. Try a system prompt that forbids the thought channel, or `--diffusion-blocks 6+`.
|
||||
2. **(Optional) Vision smoke test** — DiffusionGemma takes image+video input; only text was tested. Would need the mmproj path through `llama-diffusion-cli` / `diffusion-gemma-server`.
|
||||
3. **(Optional) Benchmark vs autoregressive gemma4:26b** on matched prompts to quantify the diffusion speed tradeoff at longer canvases (where diffusion should pull ahead).
|
||||
|
||||
### Blockers/Open Questions
|
||||
|
||||
- [ ] DiffusionGemma stays an experimental artifact until llama.cpp mainline merges the diffusion arch (then Ollama). No deployment path for Simon/AI_Visualizer/mort-bot yet.
|
||||
|
||||
### Deferred Items
|
||||
|
||||
- Did not test the `diffusion-gemma-server` (HTTP) build target — only the CLI. The server exists in PR #24423 (`examples/diffusion-gemma-server/`) if an API smoke test is wanted later.
|
||||
- Did not delete the 16.8 GB GGUF or the llama.cpp build — left at `/mnt/ai_data/diffusiongemma/` for reproducibility (325 GB free there).
|
||||
|
||||
## Context for Resuming Agent
|
||||
|
||||
### Important Context
|
||||
|
||||
The model is **not** Ollama-pullable — that's the whole point of the writeup. Anyone wanting to run it again uses `/mnt/ai_data/diffusiongemma/llama.cpp/build/bin/llama-diffusion-cli` with the Q4_K_M GGUF in the sibling `gguf/` dir. **Select the 3090 Ti by UUID** (`CUDA_VISIBLE_DEVICES=GPU-61fed72d-0698-c3b1-a789-6f93a1ed52d9`) — nvidia-smi index ≠ CUDA index on steel141 (this burned two runs). And **stop ollama first** to free VRAM, restart after.
|
||||
|
||||
### Assumptions Made
|
||||
|
||||
- CUDA toolkit 12.4 (apt) is compatible with driver 550.163.01 / CUDA 12.4 — verified, build + inference both worked.
|
||||
- The apt-installed toolkit and the 16.8 GB GGUF are fine to leave on steel141 (disk has headroom). If Seth wants them gone: `sudo apt remove nvidia-cuda-toolkit` and `rm -rf /mnt/ai_data/diffusiongemma`.
|
||||
|
||||
### Potential Gotchas
|
||||
|
||||
- **nvidia-smi GPU index ≠ CUDA device index on steel141** (3090 Ti is nvidia-smi GPU 1 but CUDA device 0). Select by UUID.
|
||||
- `/usr/bin/time` is not installed on steel141 — use shell `date +%s.%N` for timing.
|
||||
- `/mnt/ai_data` is owned by `ollama:ollama` — needed `sudo mkdir + chown claude` to write there.
|
||||
- `gh` is not installed — fetch PRs with `git fetch origin pull/N/head:branch`.
|
||||
- The `<|channel>thought` CoT eats the 256-token canvas; budget blocks or steer it.
|
||||
|
||||
## Environment State
|
||||
|
||||
### Tools/Services Used
|
||||
|
||||
- steel141 Ollama daemon (v0.30.8) — **stopped and restarted** several times during the smoke test; currently **active** and healthy (verified `curl /api/tags`).
|
||||
- 3090 Ti (nvidia-smi GPU 1 / CUDA device 0) for inference; restored to idle.
|
||||
- New: `nvidia-cuda-toolkit` 12.4.131 (`/usr/bin/nvcc`), installed this session.
|
||||
- Built binary: `/mnt/ai_data/diffusiongemma/llama.cpp/build/bin/llama-diffusion-cli`.
|
||||
|
||||
### Active Processes
|
||||
|
||||
- `ollama serve` under systemd (`systemctl is-active ollama` = active). No background jobs left from this session.
|
||||
|
||||
### Environment Variables
|
||||
|
||||
- `CUDA_VISIBLE_DEVICES` (set to the 3090 Ti UUID for runs — not persisted)
|
||||
- Ollama's `OLLAMA_HOST` / `OLLAMA_KEEP_ALIVE` (systemd override, unchanged)
|
||||
|
||||
## Related Resources
|
||||
|
||||
- `docs/reference/diffusiongemma-smoketest-2026-06-17.md` — full writeup + build recipe
|
||||
- `scripts/diffusiongemma-smoketest/run.sh` + `results/` — harness + raw logs
|
||||
- `/mnt/ai_data/diffusiongemma/` — GGUF + llama.cpp build (local, not in git)
|
||||
- Memories: `project_diffusiongemma_not_in_ollama`, `project_steel141_gpu_cuda_ordering`
|
||||
- Web: ai.google.dev/gemma/docs/diffusiongemma · huggingface.co/google/diffusiongemma-26B-A4B-it · ggml-org/llama.cpp PR #24423 · ollama/ollama#16664
|
||||
|
||||
---
|
||||
|
||||
**Security Reminder**: Before finalizing, run `validate_handoff.py` to check for accidental secret exposure.
|
||||
Reference in New Issue
Block a user