Seth/gemma4-research
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
Files
master
gemma4-research/.claude/handoffs/2026-06-17-221048-diffusiongemma-research-smoketest.md
T
Mortdecai a662df684e docs: session handoff — DiffusionGemma research + smoke test on 3090 Ti 
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-17 22:11:52 -04:00

8.7 KiB
Raw Permalink Blame History

Handoff: DiffusionGemma Research + Smoke Test (built llama-diffusion-cli on 3090 Ti)

Session Metadata

  • Created: 2026-06-17 22:10:48
  • Project: /home/claude/bin/gemma4-research
  • Branch: master
  • Session duration: ~1.5 hours

Recent Commits (for context)

  • 6bce2d4 docs: DiffusionGemma research + first-hand smoke test on 3090 Ti
  • 043a9ae docs: session handoff — gemma4:12b research + smoke test + steel141 Ollama 0.30.8 upgrade
  • 2336fb4 docs: add gemma4:12b variant + smoke-test results
  • f11aa31 docs: session handoff — Gemma 4 tool-calling May-2026 update + bot audit
  • 438a96f docs: May 2026 tool-calling update + agentic retrieval landscape

Handoff Chain

Current State Summary

Researched and smoke-tested DiffusionGemma (google/diffusiongemma-26B-A4B-it, released 2026-06-10) — Google's first open-weight text-diffusion LLM. The headline finding: it does NOT run in the homelab Ollama stack (unknown model architecture: 'diffusion-gemma') and won't until the diffusion arch merges to llama.cpp mainline. Seth chose the "build the llama.cpp fork on the 3090 Ti" path. I installed the CUDA toolkit on steel141 (was missing), built llama-diffusion-cli from ggml-org/llama.cpp PR #24423, pulled the Q4_K_M GGUF (16.8 GB), and ran a 4-prompt smoke test on the 3090 Ti. Findings committed (6bce2d4) and pushed. Work is complete — only optional follow-ups remain.

Codebase Understanding

Architecture Overview

Documentation/research corpus repo (no app code). Findings live in CORPUS_*.md + docs/reference/*.md, indexed by README.md. Smoke tests normally hit local Ollama, but DiffusionGemma needed a custom-built llama-diffusion-cli binary instead.

Critical Files

File Purpose Relevance
docs/reference/diffusiongemma-smoketest-2026-06-17.md Full writeup — specs, build recipe, throughput, gotchas Created this session
scripts/diffusiongemma-smoketest/run.sh + results/*.log Harness + raw smoke-test logs Created this session
CORPUS_ollama_variants.md Added "DiffusionGemma is NOT an Ollama variant" callout Edited this session
README.md Added reference-doc index line Edited this session

Key Patterns Discovered

  • DiffusionGemma generation: denoises a 256-token canvas in parallel via an entropy-bound decoder (13–25 steps/block, ~124 ms/step), not autoregressive. Two throughput numbers: "in-step parallel" (~2030 tok/s, canvas-parallel) vs effective (~106 tok/s end-to-end).
  • <|channel>thought CoT channel is denoised inside the canvas and eats it — strict short-format prompts truncate before the answer unless you budget --diffusion-blocks ≥ 4.

Work Completed

Tasks Finished

  • Researched DiffusionGemma (specs, modalities, run paths) — released 2026-06-10, Apache-2.0, 25.2B/3.8B-active MoE + diffusion head
  • Confirmed it does NOT run in Ollama/llama-cli (arch diffusion-gemma); needs llama-diffusion-cli from PR #24423
  • Installed nvidia-cuda-toolkit 12.4 on steel141 (nvcc was absent)
  • Built llama-diffusion-cli (CUDA sm_86) at /mnt/ai_data/diffusiongemma/llama.cpp/build/bin/
  • Downloaded Q4_K_M GGUF (16.8 GB) to /mnt/ai_data/diffusiongemma/gguf/
  • Smoke test on 3090 Ti: reasoning ✅ coherent, code ✅ correct is_prime, format ⚠️ thought-channel spiral
  • Wrote reference doc, updated corpus + README, committed (6bce2d4) + pushed
  • Saved 2 memories (DiffusionGemma-not-in-Ollama, steel141 GPU/CUDA ordering)

Files Modified

File Changes Rationale
CORPUS_ollama_variants.md "not an Ollama variant" callout block Stop anyone trying ollama pull diffusiongemma
README.md Reference-doc index row Discoverability
docs/reference/diffusiongemma-smoketest-2026-06-17.md New Full findings + build recipe
scripts/diffusiongemma-smoketest/ New harness + result logs Reproducibility

Decisions Made

Decision Options Considered Rationale
llama.cpp fork on 3090 Ti (a) fork+GGUF on 3090 Ti, (b) Transformers 4-bit, (c) cloud H100 vLLM Seth chose (a) — local, free, closest to the GGUF path he'd deploy on once it merges
Install CUDA toolkit via apt manual CUDA tarball apt 12.4 matches the 550 driver's CUDA 12.4 exactly; reversible
Stop ollama per run window ollama stop <model> only stop-model loses the race to live traffic that re-loads gemma4:26b; full service stop is deterministic, bounded ~3 min

Pending Work

Immediate Next Steps

  1. (Optional) Re-run with thinking suppressed / more blocks to get clean short-format outputs — the <|channel>thought channel needs steering. Try a system prompt that forbids the thought channel, or --diffusion-blocks 6+.
  2. (Optional) Vision smoke test — DiffusionGemma takes image+video input; only text was tested. Would need the mmproj path through llama-diffusion-cli / diffusion-gemma-server.
  3. (Optional) Benchmark vs autoregressive gemma4:26b on matched prompts to quantify the diffusion speed tradeoff at longer canvases (where diffusion should pull ahead).

Blockers/Open Questions

  • DiffusionGemma stays an experimental artifact until llama.cpp mainline merges the diffusion arch (then Ollama). No deployment path for Simon/AI_Visualizer/mort-bot yet.

Deferred Items

  • Did not test the diffusion-gemma-server (HTTP) build target — only the CLI. The server exists in PR #24423 (examples/diffusion-gemma-server/) if an API smoke test is wanted later.
  • Did not delete the 16.8 GB GGUF or the llama.cpp build — left at /mnt/ai_data/diffusiongemma/ for reproducibility (325 GB free there).

Context for Resuming Agent

Important Context

The model is not Ollama-pullable — that's the whole point of the writeup. Anyone wanting to run it again uses /mnt/ai_data/diffusiongemma/llama.cpp/build/bin/llama-diffusion-cli with the Q4_K_M GGUF in the sibling gguf/ dir. Select the 3090 Ti by UUID (CUDA_VISIBLE_DEVICES=GPU-61fed72d-0698-c3b1-a789-6f93a1ed52d9) — nvidia-smi index ≠ CUDA index on steel141 (this burned two runs). And stop ollama first to free VRAM, restart after.

Assumptions Made

  • CUDA toolkit 12.4 (apt) is compatible with driver 550.163.01 / CUDA 12.4 — verified, build + inference both worked.
  • The apt-installed toolkit and the 16.8 GB GGUF are fine to leave on steel141 (disk has headroom). If Seth wants them gone: sudo apt remove nvidia-cuda-toolkit and rm -rf /mnt/ai_data/diffusiongemma.

Potential Gotchas

  • nvidia-smi GPU index ≠ CUDA device index on steel141 (3090 Ti is nvidia-smi GPU 1 but CUDA device 0). Select by UUID.
  • /usr/bin/time is not installed on steel141 — use shell date +%s.%N for timing.
  • /mnt/ai_data is owned by ollama:ollama — needed sudo mkdir + chown claude to write there.
  • gh is not installed — fetch PRs with git fetch origin pull/N/head:branch.
  • The <|channel>thought CoT eats the 256-token canvas; budget blocks or steer it.

Environment State

Tools/Services Used

  • steel141 Ollama daemon (v0.30.8) — stopped and restarted several times during the smoke test; currently active and healthy (verified curl /api/tags).
  • 3090 Ti (nvidia-smi GPU 1 / CUDA device 0) for inference; restored to idle.
  • New: nvidia-cuda-toolkit 12.4.131 (/usr/bin/nvcc), installed this session.
  • Built binary: /mnt/ai_data/diffusiongemma/llama.cpp/build/bin/llama-diffusion-cli.

Active Processes

  • ollama serve under systemd (systemctl is-active ollama = active). No background jobs left from this session.

Environment Variables

  • CUDA_VISIBLE_DEVICES (set to the 3090 Ti UUID for runs — not persisted)
  • Ollama's OLLAMA_HOST / OLLAMA_KEEP_ALIVE (systemd override, unchanged)

Related Resources

  • docs/reference/diffusiongemma-smoketest-2026-06-17.md — full writeup + build recipe
  • scripts/diffusiongemma-smoketest/run.sh + results/ — harness + raw logs
  • /mnt/ai_data/diffusiongemma/ — GGUF + llama.cpp build (local, not in git)
  • Memories: project_diffusiongemma_not_in_ollama, project_steel141_gpu_cuda_ordering
  • Web: ai.google.dev/gemma/docs/diffusiongemma · huggingface.co/google/diffusiongemma-26B-A4B-it · ggml-org/llama.cpp PR #24423 · ollama/ollama#16664

Security Reminder: Before finalizing, run validate_handoff.py to check for accidental secret exposure.

Reference in New Issue View Git Blame Copy Permalink