gemma4-research/.claude/handoffs/2026-06-17-221048-diffusiongemma-research-smoketest.md

# Handoff: DiffusionGemma Research + Smoke Test (built llama-diffusion-cli on 3090 Ti)

## Session Metadata
- Created: 2026-06-17 22:10:48
- Project: /home/claude/bin/gemma4-research
- Branch: master
- Session duration: ~1.5 hours

### Recent Commits (for context)
  - 6bce2d4 docs: DiffusionGemma research + first-hand smoke test on 3090 Ti
  - 043a9ae docs: session handoff — gemma4:12b research + smoke test + steel141 Ollama 0.30.8 upgrade
  - 2336fb4 docs: add gemma4:12b variant + smoke-test results
  - f11aa31 docs: session handoff — Gemma 4 tool-calling May-2026 update + bot audit
  - 438a96f docs: May 2026 tool-calling update + agentic retrieval landscape

## Handoff Chain

- **Continues from**: [2026-06-16-230331-gemma4-12b-research-smoketest.md](./2026-06-16-230331-gemma4-12b-research-smoketest.md)
  - Previous title: gemma4:12b Research + Smoke Test (and steel141 Ollama 0.20.4→0.30.8 upgrade)
- **Supersedes**: None

## Current State Summary

Researched and smoke-tested **DiffusionGemma** (`google/diffusiongemma-26B-A4B-it`, released 2026-06-10) — Google's first open-weight **text-diffusion** LLM. The headline finding: it does **NOT** run in the homelab Ollama stack (`unknown model architecture: 'diffusion-gemma'`) and won't until the diffusion arch merges to llama.cpp mainline. Seth chose the "build the llama.cpp fork on the 3090 Ti" path. I installed the CUDA toolkit on steel141 (was missing), built `llama-diffusion-cli` from ggml-org/llama.cpp **PR #24423**, pulled the Q4_K_M GGUF (16.8 GB), and ran a 4-prompt smoke test on the 3090 Ti. Findings committed (6bce2d4) and pushed. Work is **complete** — only optional follow-ups remain.

## Codebase Understanding

### Architecture Overview

Documentation/research corpus repo (no app code). Findings live in `CORPUS_*.md` + `docs/reference/*.md`, indexed by `README.md`. Smoke tests normally hit local Ollama, but DiffusionGemma needed a custom-built `llama-diffusion-cli` binary instead.

### Critical Files

| File | Purpose | Relevance |
|------|---------|-----------|
| `docs/reference/diffusiongemma-smoketest-2026-06-17.md` | **Full writeup** — specs, build recipe, throughput, gotchas | Created this session |
| `scripts/diffusiongemma-smoketest/run.sh` + `results/*.log` | Harness + raw smoke-test logs | Created this session |
| `CORPUS_ollama_variants.md` | Added "DiffusionGemma is NOT an Ollama variant" callout | Edited this session |
| `README.md` | Added reference-doc index line | Edited this session |

### Key Patterns Discovered

- **DiffusionGemma generation**: denoises a 256-token canvas in parallel via an entropy-bound decoder (13–25 steps/block, ~124 ms/step), not autoregressive. Two throughput numbers: "in-step parallel" (~2030 tok/s, canvas-parallel) vs **effective** (~106 tok/s end-to-end).
- **`<|channel>thought` CoT channel** is denoised *inside* the canvas and eats it — strict short-format prompts truncate before the answer unless you budget `--diffusion-blocks` ≥ 4.

## Work Completed

### Tasks Finished

- [x] Researched DiffusionGemma (specs, modalities, run paths) — released 2026-06-10, Apache-2.0, 25.2B/3.8B-active MoE + diffusion head
- [x] Confirmed it does NOT run in Ollama/llama-cli (arch `diffusion-gemma`); needs `llama-diffusion-cli` from PR #24423
- [x] Installed `nvidia-cuda-toolkit` 12.4 on steel141 (nvcc was absent)
- [x] Built `llama-diffusion-cli` (CUDA sm_86) at `/mnt/ai_data/diffusiongemma/llama.cpp/build/bin/`
- [x] Downloaded Q4_K_M GGUF (16.8 GB) to `/mnt/ai_data/diffusiongemma/gguf/`
- [x] Smoke test on 3090 Ti: reasoning ✅ coherent, code ✅ correct `is_prime`, format ⚠️ thought-channel spiral
- [x] Wrote reference doc, updated corpus + README, committed (6bce2d4) + pushed
- [x] Saved 2 memories (DiffusionGemma-not-in-Ollama, steel141 GPU/CUDA ordering)

### Files Modified

| File | Changes | Rationale |
|------|---------|-----------|
| `CORPUS_ollama_variants.md` | "not an Ollama variant" callout block | Stop anyone trying `ollama pull diffusiongemma` |
| `README.md` | Reference-doc index row | Discoverability |
| `docs/reference/diffusiongemma-smoketest-2026-06-17.md` | New | Full findings + build recipe |
| `scripts/diffusiongemma-smoketest/` | New harness + result logs | Reproducibility |

### Decisions Made

| Decision | Options Considered | Rationale |
|----------|-------------------|-----------|
| llama.cpp fork on 3090 Ti | (a) fork+GGUF on 3090 Ti, (b) Transformers 4-bit, (c) cloud H100 vLLM | Seth chose (a) — local, free, closest to the GGUF path he'd deploy on once it merges |
| Install CUDA toolkit via apt | manual CUDA tarball | apt 12.4 matches the 550 driver's CUDA 12.4 exactly; reversible |
| Stop ollama per run window | `ollama stop <model>` only | stop-model loses the race to live traffic that re-loads gemma4:26b; full service stop is deterministic, bounded ~3 min |

## Pending Work

### Immediate Next Steps

1. **(Optional) Re-run with thinking suppressed / more blocks** to get clean short-format outputs — the `<|channel>thought` channel needs steering. Try a system prompt that forbids the thought channel, or `--diffusion-blocks 6+`.
2. **(Optional) Vision smoke test** — DiffusionGemma takes image+video input; only text was tested. Would need the mmproj path through `llama-diffusion-cli` / `diffusion-gemma-server`.
3. **(Optional) Benchmark vs autoregressive gemma4:26b** on matched prompts to quantify the diffusion speed tradeoff at longer canvases (where diffusion should pull ahead).

### Blockers/Open Questions

- [ ] DiffusionGemma stays an experimental artifact until llama.cpp mainline merges the diffusion arch (then Ollama). No deployment path for Simon/AI_Visualizer/mort-bot yet.

### Deferred Items

- Did not test the `diffusion-gemma-server` (HTTP) build target — only the CLI. The server exists in PR #24423 (`examples/diffusion-gemma-server/`) if an API smoke test is wanted later.
- Did not delete the 16.8 GB GGUF or the llama.cpp build — left at `/mnt/ai_data/diffusiongemma/` for reproducibility (325 GB free there).

## Context for Resuming Agent

### Important Context

The model is **not** Ollama-pullable — that's the whole point of the writeup. Anyone wanting to run it again uses `/mnt/ai_data/diffusiongemma/llama.cpp/build/bin/llama-diffusion-cli` with the Q4_K_M GGUF in the sibling `gguf/` dir. **Select the 3090 Ti by UUID** (`CUDA_VISIBLE_DEVICES=GPU-61fed72d-0698-c3b1-a789-6f93a1ed52d9`) — nvidia-smi index ≠ CUDA index on steel141 (this burned two runs). And **stop ollama first** to free VRAM, restart after.

### Assumptions Made

- CUDA toolkit 12.4 (apt) is compatible with driver 550.163.01 / CUDA 12.4 — verified, build + inference both worked.
- The apt-installed toolkit and the 16.8 GB GGUF are fine to leave on steel141 (disk has headroom). If Seth wants them gone: `sudo apt remove nvidia-cuda-toolkit` and `rm -rf /mnt/ai_data/diffusiongemma`.

### Potential Gotchas

- **nvidia-smi GPU index ≠ CUDA device index on steel141** (3090 Ti is nvidia-smi GPU 1 but CUDA device 0). Select by UUID.
- `/usr/bin/time` is not installed on steel141 — use shell `date +%s.%N` for timing.
- `/mnt/ai_data` is owned by `ollama:ollama` — needed `sudo mkdir + chown claude` to write there.
- `gh` is not installed — fetch PRs with `git fetch origin pull/N/head:branch`.
- The `<|channel>thought` CoT eats the 256-token canvas; budget blocks or steer it.

## Environment State

### Tools/Services Used

- steel141 Ollama daemon (v0.30.8) — **stopped and restarted** several times during the smoke test; currently **active** and healthy (verified `curl /api/tags`).
- 3090 Ti (nvidia-smi GPU 1 / CUDA device 0) for inference; restored to idle.
- New: `nvidia-cuda-toolkit` 12.4.131 (`/usr/bin/nvcc`), installed this session.
- Built binary: `/mnt/ai_data/diffusiongemma/llama.cpp/build/bin/llama-diffusion-cli`.

### Active Processes

- `ollama serve` under systemd (`systemctl is-active ollama` = active). No background jobs left from this session.

### Environment Variables

- `CUDA_VISIBLE_DEVICES` (set to the 3090 Ti UUID for runs — not persisted)
- Ollama's `OLLAMA_HOST` / `OLLAMA_KEEP_ALIVE` (systemd override, unchanged)

## Related Resources

- `docs/reference/diffusiongemma-smoketest-2026-06-17.md` — full writeup + build recipe
- `scripts/diffusiongemma-smoketest/run.sh` + `results/` — harness + raw logs
- `/mnt/ai_data/diffusiongemma/` — GGUF + llama.cpp build (local, not in git)
- Memories: `project_diffusiongemma_not_in_ollama`, `project_steel141_gpu_cuda_ordering`
- Web: ai.google.dev/gemma/docs/diffusiongemma · huggingface.co/google/diffusiongemma-26B-A4B-it · ggml-org/llama.cpp PR #24423 · ollama/ollama#16664

---

**Security Reminder**: Before finalizing, run `validate_handoff.py` to check for accidental secret exposure.