gemma4-research/scripts/native-bakeoff/README.md

Native Bakeoff — Gemma 4 Inference Path Comparison

Three-arm bakeoff comparing how different inference paths handle the same Gemma 4 tool-calling workload. Isolates Ollama's JSON↔native translator and the runtime itself as variables.

The three arms

Arm	Path	What varies
A. ollama-json	/api/chat with OpenAI-style tools:[...]	Ollama translates JSON → native tokens on input, native tool-call tokens → structured JSON on output.
B. ollama-native	/api/generate with raw:true + canonical HF jinja template	No JSON translation. Rendered tokens go straight to the model; the harness parses <|tool_call> spans out of the completion.
C. jax-native	google-deepmind/gemma reference ToolSampler	No Ollama. No llama.cpp. No GGUF quant. Reference Python + JAX + bf16.

Research question

Does Ollama's JSON tools path materially diverge from the native/reference path?

• A vs B divergence ⇒ the Ollama server-side parser is the variable.
• B vs C divergence ⇒ llama.cpp runtime / GGUF quantization / Ollama scheduler is the variable.
• A ≡ B ≡ C ⇒ Ollama's path is faithful to the reference, current production usage is fine.

Prerequisites

Arms A and B: local Ollama with gemma4:latest (E4B 8B) or gemma4:e4b-it-q8_0 pulled. Python 3.10+, aiohttp, jinja2.

Arm C: separate env with jax and gemma installed; HF credentials for checkpoint download (~8GB for E4B-it). See arms/jax_native.py module docstring.

Running

cd scripts/native-bakeoff

# One arm, one task:
python3 harness.py --arm ollama-json   --task memory --out runs/A/memory.json
python3 harness.py --arm ollama-native --task memory --out runs/B/memory.json
python3 harness.py --arm jax-native    --task memory --out runs/C/memory.json

# Full sweep (A + B, 4 tasks each):
for arm in ollama-json ollama-native; do
  for task in movies research memory long; do
    python3 harness.py --arm "$arm" --task "$task" \
      --out "runs/${arm}/${task}.json"
  done
done

Default model is gemma4:latest for Ollama arms (the E4B-it variant). Override with --model gemma4:26b if you want the MoE bakeoff (expect slower; 26B is 18GB GGUF).

Trace schema

Each run writes a JSON with:

• arm, model, task, task_prompt
• turns[] — per-step metrics: elapsed_s, prompt_eval_count, eval_count, tool_call_count, content_len, etc.
• final — halt_reason, steps_used, tool_calls_total, wall_clock_s, final_history_chars

Halt reasons: no_tool_calls (model produced final answer), step_budget (hit 20-step limit), error:*, env_missing (arm C only), sampler_error:* (arm C only).

Smoke test evidence

First wiring run on 2026-04-19 against gemma4:latest on steel141 (local Ollama, CPU):

Arm	Task	Steps	Tools	Halt	Wall
A (ollama-json)	memory	2	1	no_tool_calls	10.16s
B (ollama-native)	memory	2	1	no_tool_calls	2.39s

Identical behavioral shape (one tool call, clean final answer) on this simple task. The wall-clock delta is interesting but not conclusive on a single run — could be cache warmth or could be Ollama's parser overhead. A full sweep will separate signal from noise.

Known limitations

• Arm C system prompt handling. gm.text.ToolSampler doesn't take a pre-populated message history cleanly, so arm C folds a compact version of FAKE_HISTORY into the user message. Arms A and B feed history through proper role-tagged turns. Fidelity compromise — if a C vs A/B delta traces here, rebuild sampler.turns directly before calling .chat().
• Arm C sampler caveat. The deepmind-gemma ToolSampler docstring notes "Gemma 1, 2 and 3 models were not specifically trained for tool use" and flags the sampler as a proof-of-concept. Gemma 4 is tool-trained, so it should do better, but if arm C underperforms A/B the sampler implementation may be the variable, not the model.
• Quantization confounder. Ollama arms run Q8 (E4B) or Q4 (26B); arm C runs bf16. A non-trivial A vs C delta could be the quantization. Only A ≡ B ≢ C cleanly implicates the inference engine rather than the bits.

Related artifacts

• scripts/mort-bakeoff/harness.py — the round-3 bakeoff that established think:false kills 26B in multi-turn tool loops. Task definitions are lifted from there.
• docs/reference/bakeoff-2026-04-18.md — round-3 writeup.
• CORPUS_tool_calling_format.md — the native Gemma 4 tool-call token syntax this harness implements.
• tooling/huggingface/model-cards/gemma-4-E4B-it-chat_template.jinja — the canonical template arm B renders.