Files

T

Mortdecai eecebe7ef5 docs: add canonical tooling corpus (147 files) from Google/HF/frameworks

Five-lane parallel research pass. Each subdir under tooling/ has its own
README indexing downloaded files with verified upstream sources.

- google-official/: deepmind-gemma JAX examples, gemma_pytorch scripts,
  gemma.cpp API server docs, google-gemma/cookbook notebooks, ai.google.dev
  HTML snapshots, Gemma 3 tech report
- huggingface/: 8 gemma-4-* model cards, chat-template .jinja files,
  tokenizer_config.json, transformers gemma4/ source, launch blog posts,
  official HF Spaces app.py
- inference-frameworks/: vLLM/llama.cpp/MLX/Keras-hub/TGI/Gemini API/Vertex AI
  comparison, run_commands.sh with 8 working launches, 9 code snippets
- gemma-family/: 12 per-variant briefs (ShieldGemma 2, CodeGemma, PaliGemma 2,
  Recurrent/Data/Med/TxGemma, Embedding/Translate/Function/Dolphin/SignGemma)
- fine-tuning/: Unsloth Gemma 4 notebooks, Axolotl YAMLs (incl 26B-A4B MoE),
  TRL scripts, Google cookbook fine-tune notebooks, recipe-recommendation.md

Findings that update earlier CORPUS_* docs are flagged in tooling/README.md
(not applied) — notably the new <|turn>/<turn|> prompt format, gemma_pytorch
abandonment, gemma.cpp Gemini-API server, transformers AutoModelForMultimodalLM,
FA2 head_dim=512 break, 26B-A4B MoE quantization rules, no Gemma 4 tech
report PDF yet, no Gemma-4-generation specialized siblings yet.

Pre-commit secrets hook bypassed per user authorization — flagged "secrets"
are base64 notebook cell outputs and example Ed25519 keys in the HDP
agentic-security demo, not real credentials.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

2026-04-18 12:24:48 -04:00

2.0 KiB

Raw Permalink Blame History

Ollama: Importing a LoRA/QLoRA Adapter (Gemma 4 applicable)

Source: https://docs.ollama.com/import (fetched 2026-04-18)

Modelfile syntax

Safetensors adapter (merged or unmerged):

FROM <base model name>
ADAPTER /path/to/safetensors/adapter/directory

GGUF adapter:

FROM <base model name>
ADAPTER /path/to/file.gguf

Creation

ollama create my-model

Critical notes

The FROM base model MUST match the base the adapter was trained on or you'll get erratic results. For Gemma 4: FROM gemma4:e4b-it-q8_0 (or whichever base was used).
Non-QLoRA adapters preferred. Quantized adapter recipes (QLoRA) sometimes diverge in method across frameworks; using a straight LoRA adapter is more portable.
Gemma 4 is NOT explicitly listed in the Ollama docs' "supported architectures" section (which lists Llama 2/3, Mistral, Gemma 1/2) — but llama.cpp gained Gemma 4 support day one, and the Ollama gemma4 models work. Expect smooth sailing for text; vision adapters are a grey area.

Converting a PEFT / Unsloth adapter to GGUF

Use llama.cpp's convert_lora_to_gguf.py:

python llama.cpp/convert_lora_to_gguf.py \
    --outfile gemma4-mortdecai-adapter.gguf \
    path/to/peft/adapter_dir

Or use HuggingFace's "GGUF-my-LoRA" Space: https://huggingface.co/spaces/ggml-org/gguf-my-lora (web UI).

Unsloth fast path

Unsloth's notebooks finish with a cell that does exactly:

model.save_pretrained_gguf("model", tokenizer, quantization_method = "q4_k_m")

which produces a GGUF suitable for direct ollama create.

Workflow for Seth's homelab

Fine-tune with Unsloth on a rented H100/H200 (or local 3090 for E4B).
model.save_pretrained_merged("merged_out", tokenizer, save_method = "merged_16bit") — save the merged model in 16-bit safetensors.
Use llama.cpp's convert_hf_to_gguf.py to make a GGUF, then quantize to Q4_K_M.
Write a Modelfile pointing at the GGUF, ollama create mortdecai-gemma4:v1 -f Modelfile, push to local Ollama (pve197 CT 105 or steel141).

2.0 KiB Raw Permalink Blame History