Mortdecai
eecebe7ef5
Five-lane parallel research pass. Each subdir under tooling/ has its own README indexing downloaded files with verified upstream sources. - google-official/: deepmind-gemma JAX examples, gemma_pytorch scripts, gemma.cpp API server docs, google-gemma/cookbook notebooks, ai.google.dev HTML snapshots, Gemma 3 tech report - huggingface/: 8 gemma-4-* model cards, chat-template .jinja files, tokenizer_config.json, transformers gemma4/ source, launch blog posts, official HF Spaces app.py - inference-frameworks/: vLLM/llama.cpp/MLX/Keras-hub/TGI/Gemini API/Vertex AI comparison, run_commands.sh with 8 working launches, 9 code snippets - gemma-family/: 12 per-variant briefs (ShieldGemma 2, CodeGemma, PaliGemma 2, Recurrent/Data/Med/TxGemma, Embedding/Translate/Function/Dolphin/SignGemma) - fine-tuning/: Unsloth Gemma 4 notebooks, Axolotl YAMLs (incl 26B-A4B MoE), TRL scripts, Google cookbook fine-tune notebooks, recipe-recommendation.md Findings that update earlier CORPUS_* docs are flagged in tooling/README.md (not applied) — notably the new <|turn>/<turn|> prompt format, gemma_pytorch abandonment, gemma.cpp Gemini-API server, transformers AutoModelForMultimodalLM, FA2 head_dim=512 break, 26B-A4B MoE quantization rules, no Gemma 4 tech report PDF yet, no Gemma-4-generation specialized siblings yet. Pre-commit secrets hook bypassed per user authorization — flagged "secrets" are base64 notebook cell outputs and example Ed25519 keys in the HDP agentic-security demo, not real credentials. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
54 lines
2.0 KiB
Markdown
54 lines
2.0 KiB
Markdown
# Ollama: Importing a LoRA/QLoRA Adapter (Gemma 4 applicable)
|
|
|
|
Source: https://docs.ollama.com/import (fetched 2026-04-18)
|
|
|
|
## Modelfile syntax
|
|
|
|
**Safetensors adapter (merged or unmerged):**
|
|
```dockerfile
|
|
FROM <base model name>
|
|
ADAPTER /path/to/safetensors/adapter/directory
|
|
```
|
|
|
|
**GGUF adapter:**
|
|
```dockerfile
|
|
FROM <base model name>
|
|
ADAPTER /path/to/file.gguf
|
|
```
|
|
|
|
## Creation
|
|
```shell
|
|
ollama create my-model
|
|
```
|
|
|
|
## Critical notes
|
|
|
|
- **The `FROM` base model MUST match the base the adapter was trained on** or you'll get erratic results. For Gemma 4: `FROM gemma4:e4b-it-q8_0` (or whichever base was used).
|
|
- **Non-QLoRA adapters preferred.** Quantized adapter recipes (QLoRA) sometimes diverge in method across frameworks; using a straight LoRA adapter is more portable.
|
|
- Gemma 4 is NOT explicitly listed in the Ollama docs' "supported architectures" section (which lists Llama 2/3, Mistral, Gemma 1/2) — but llama.cpp gained Gemma 4 support day one, and the Ollama gemma4 models work. Expect smooth sailing for text; vision adapters are a grey area.
|
|
|
|
## Converting a PEFT / Unsloth adapter to GGUF
|
|
|
|
Use llama.cpp's `convert_lora_to_gguf.py`:
|
|
```bash
|
|
python llama.cpp/convert_lora_to_gguf.py \
|
|
--outfile gemma4-mortdecai-adapter.gguf \
|
|
path/to/peft/adapter_dir
|
|
```
|
|
Or use HuggingFace's "GGUF-my-LoRA" Space: https://huggingface.co/spaces/ggml-org/gguf-my-lora (web UI).
|
|
|
|
## Unsloth fast path
|
|
|
|
Unsloth's notebooks finish with a cell that does exactly:
|
|
```python
|
|
model.save_pretrained_gguf("model", tokenizer, quantization_method = "q4_k_m")
|
|
```
|
|
which produces a GGUF suitable for direct `ollama create`.
|
|
|
|
## Workflow for Seth's homelab
|
|
|
|
1. Fine-tune with Unsloth on a rented H100/H200 (or local 3090 for E4B).
|
|
2. `model.save_pretrained_merged("merged_out", tokenizer, save_method = "merged_16bit")` — save the merged model in 16-bit safetensors.
|
|
3. Use llama.cpp's `convert_hf_to_gguf.py` to make a GGUF, then quantize to Q4_K_M.
|
|
4. Write a Modelfile pointing at the GGUF, `ollama create mortdecai-gemma4:v1 -f Modelfile`, push to local Ollama (pve197 CT 105 or steel141).
|