feat: mort-bot think=true vs think=false bakeoff

Seth's challenge: "we experienced this context eating with every
implementation that had think=true. mort-bot runs a loop. Can you do
a bake-off?"

Built a harness that replicates mort-bot's /api/chat loop verbatim
(num_ctx=8192, num_predict=2048, temperature=0.7, gemma4:26b,
STEP_BUDGET=20, exact payload shape) but with stubbed tools and a
prebuilt 15-turn fake chat history. Ran 4 tasks × 2 think settings.

Finding: on Ollama 0.20.4 the "thinking eats context" concern does
NOT reproduce. Direct evidence:
- Movies task step 2 (think=true) returned 905 chars of thinking.
- Step 3 prompt_eval_count delta: +76 tokens (think=true) vs +135
  tokens (think=false). If thinking had accumulated in the prompt,
  think=true would have grown by +360 tokens, not shrunk.
- Ollama's chat template strips the `thinking` field when serializing
  assistant turns for subsequent prompts.

All 4 tasks × 2 settings produced identical step counts and tool
counts. Wall clocks comparable. Gemma only actually generated
thinking on 1 of 4 tasks (the one with check_sethflix verify-loop);
on the others with think=true it emitted 0 thinking tokens.

Reconciled with the earlier coding-agent bakeoff: the two findings
are orthogonal. Coding bakeoff was at num_ctx=32K with a different
harness; mort at 8K doesn't touch the silent-stop regime either way.
Seth's prior may have been correct on an older Ollama or in a
different API shape (/api/generate has its own issues) but does not
reproduce here.

Concrete recommendation: mort-bot THINK=False is defensible but not
load-bearing; THINK=True or unset-default would also work. Keep as-is
unless a different need arises.

New: docs/reference/mort-bakeoff-2026-04-18.md, scripts/mort-bakeoff/
(harness + 8 run logs). README updated with pointer.

.secrets.baseline

@@ -14579,5 +14579,5 @@
       }
     ]
   },
-  "generated_at": "2026-04-18T22:14:05Z"
+  "generated_at": "2026-04-18T22:23:43Z"
 }

README.md

@@ -15,7 +15,8 @@ Research corpus and implementation guidance for Google Gemma 4, based on product
 | `CORPUS_benchmarks.md` | Full benchmark table vs Gemma 3, arena scores, agentic scores | When comparing Gemma 4 to alternatives |
 | `CORPUS_tool_calling_format.md` | Native token format + JSON API format for function calling | When implementing tool calling |
 | `CORPUS_cli_coding_agent.md` | Positioning Gemma 4 for CLI coding agent use (openclaw / open code / pi / hermes / aider style). Honest take on what Google did and didn't measure, head-to-head with `qwen3-coder:30b`, homelab setup pointer | When scoping a CLI coding agent or deciding Gemma 4 vs Qwen3-Coder |
-| `docs/reference/bakeoff-2026-04-18.md` | Raw results: CLI-coding-agent bakeoff of gemma4:26b / gemma4:31b / qwen3-coder:30b on steel141 3090 Ti. **31B clean, Qwen3-Coder correct but chatty, 26B reproducibly silent-stops at write_file.** Harness at `scripts/bakeoff/` | When deciding which model to back a CLI agent with, or debugging a similar tool-call halt |
+| `docs/reference/bakeoff-2026-04-18.md` | CLI-coding-agent bakeoff on 3090 Ti. **Rounds 1/2 misidentified the cause; Round 3 (the correct one): `think: false` silent-stops gemma4:26b at certain multi-turn states on 32K context.** 31B and Qwen3-Coder robust to the flag. Harness at `scripts/bakeoff/` | When deciding which model to back a CLI agent with, writing a custom agent payload, or debugging a silent tool-call halt |
+| `docs/reference/mort-bakeoff-2026-04-18.md` | mort-bot-specific `think=true` vs `think=false` bakeoff on mort's actual loop shape (gemma4:26b, num_ctx=8192). **Thinking does NOT accumulate in context on Ollama 0.20.4** — strips it from serialized history. Both settings behave identically on step counts, tool counts, wall clock. Harness at `scripts/mort-bakeoff/` | When deciding mort-bot's THINK env var, or when someone claims "think=true eats context" without pinning an Ollama version |
 | `tooling/` | **Canonical upstream tooling** — real scripts, notebooks, model cards, and configs pulled from Google / HF / framework maintainers (147 files). Subdirs: `google-official/`, `huggingface/`, `inference-frameworks/`, `gemma-family/`, `fine-tuning/`. See `tooling/README.md` for index and findings that update the older `CORPUS_*` docs | When you need authoritative source material — model cards, chat templates, fine-tuning recipes, serving commands for vLLM / llama.cpp / MLX, or to scope a specialized sibling (ShieldGemma, EmbeddingGemma, etc.) |
 
 ## Source Projects

docs/reference/mort-bakeoff-2026-04-18.md

@@ -0,0 +1,155 @@
+# mort-bot `think=true` vs `think=false` Bakeoff — 2026-04-18
+
+> Follow-up to Seth's challenge: "we experienced this context eating with every
+> implementation that had think=true. mort-bot runs a loop. Can you do a
+> bake-off to see if that bot would actually perform better with thinking on?"
+>
+> Short answer: **no measurable difference on Ollama 0.20.4**. The
+> "thinking-eats-context" concern doesn't reproduce in mort-bot's current
+> loop shape because Ollama's chat template strips the `thinking` field from
+> serialized history when it builds the prompt for subsequent turns. Either
+> setting is defensible.
+
+## Setup
+
+- Harness: `scripts/mort-bakeoff/harness.py` — replicates mort-bot `llm.py`
+  `run_tool_loop` call shape verbatim (model, options, payload structure,
+  `messages.append(msg)` behavior), but with stubbed tools and a prebuilt
+  ~15-turn fake chat history to simulate mid-session state.
+- Host / Ollama: steel141, 3090 Ti, Ollama 0.20.4
+- Exact config match to mort-bot production: `gemma4:26b`, `num_ctx=8192`,
+  `num_predict=2048`, `temperature=0.7`, `top_p=0.95`, `top_k=64`,
+  `keep_alive=2h`, `STEP_BUDGET=20`
+- Tasks: 4 scenarios representative of real traffic (movies, research,
+  memory, long-chain research)
+- n=1 per (task, think) cell — bakeoff, not benchmark
+
+## Results
+
+| Task | Think | Steps | Tools | Peak prompt | Thinking generated | Wall |
+|---|---|---|---|---|---|---|
+| memory | false | 2 | 1 | 1421 | 0 tok | 1.7s |
+| memory | true | 2 | 1 | 1422 | 0 tok | 2.0s |
+| research | false | 2 | 2 | 1593 | 0 tok | 2.3s |
+| research | true | 2 | 2 | 1594 | 0 tok | 2.3s |
+| movies | false | 3 | 2 | 1635 | 0 tok | 8.3s |
+| movies | true | 3 | 2 | 1577 | 905 chars / ~226 tok | 5.4s |
+| long | false | 7 | 6 | 2243 | 0 tok | 7.8s |
+| long | true | 7 | 6 | 2288 | 0 tok | 8.0s |
+
+Every (task, think) pair produced **identical step counts and tool counts**.
+
+## Does thinking accumulate in context?
+
+**No — verified directly.**
+
+On the `movies` task, step 2 with `think=true` returned 905 chars (~226 tok)
+in a separate `thinking` field. My harness then appends the full message dict
+(including `thinking`) to `messages` before the next request — exactly what
+mort-bot's `ollama_messages.append(msg)` does.
+
+Step 2 → Step 3 prompt_eval_count delta:
+
+| Setting | Delta | What that means |
+|---|---|---|
+| think=false | +135 tok | Tool result only |
+| think=true | +76 tok | Tool result only — thinking was **stripped** |
+
+If thinking had accumulated, the think=true delta would have been ~360 tokens
+(tool result + thinking). Instead it was smaller than think=false's delta.
+**Ollama 0.20.4's chat template does not include the `thinking` field when
+serializing an assistant turn for subsequent prompts.** Thinking is a per-turn
+response annotation, not a persisted conversation channel.
+
+## Does thinking actually happen?
+
+Gemma 4 is conservative. With `think=true`, it chose to generate thinking
+tokens on **1 of 4 tasks**:
+
+- `memory` — 0 thinking tokens (simple lookup)
+- `research` — 0 thinking tokens (clear sequential plan)
+- `long` — 0 thinking tokens across 7 steps (explicit step list to follow)
+- `movies` — 905 chars thinking on step 2 (the only task requiring
+  verification logic: check candidates → filter IN LIBRARY → replace → re-check)
+
+The model appears to decide whether to think based on whether the task has
+uncertainty to reason about. Following explicit multi-step instructions
+doesn't trigger it; generating + filtering recommendations does.
+
+## Answer to Seth's claim
+
+Seth's prior: "we experienced this context eating with every implementation
+that had think=true."
+
+On current Ollama (0.20.4) with mort-bot's loop shape: **the claim does not
+reproduce.** Possibilities for why Seth saw it before:
+
+1. **Older Ollama version.** Earlier 0.x releases may have serialized `thinking`
+   into subsequent prompts. Not tested here.
+2. **Different API shape.** `/api/generate` behaves differently from `/api/chat`
+   — mort-bot's own CLAUDE.md notes `/api/generate` with `think=true` returns
+   empty responses. That's a separate, real bug, not a context-growth bug.
+3. **AI_Visualizer-shaped pipelines** generate into `content` where thinking
+   tokens explicitly eat the `num_predict` budget. That is a real failure mode
+   and the original "always think:false" guidance addresses it correctly.
+4. **Confounded by other issues** — context truncation, model quirks, silent
+   failures — misattributed to thinking.
+
+The production `THINK=False` setting was defensible when adopted and remains
+defensible now. It's just not load-bearing in the way the original guidance
+suggested.
+
+## Concrete recommendation for mort-bot
+
+1. **Keep `THINK=False` as-is, or try unset-default, or `THINK=True` — pick based on whether you want the slight quality hedge on reasoning-heavy turns (like the movies check_sethflix verify-loop). No context-growth penalty either way on 0.20.4.**
+2. **Don't backport the "CLI coding agent" finding from `docs/reference/bakeoff-2026-04-18.md`** — that one was at `num_ctx=32768` with a coding harness, different regime. Mort's `num_ctx=8192` doesn't touch the silent-stop trigger.
+3. **If Ollama versions drift**, re-run this harness. The stripping behavior is an Ollama implementation detail; a future version could change it.
+
+## Reconciling with the earlier coding-agent bakeoff
+
+The two findings are orthogonal:
+
+| Bakeoff | Context | Harness | think=false | think=true |
+|---|---|---|---|---|
+| CLI coding (Round 3) | 32K | custom coding loop | 26B silent-stops | works |
+| mort-bot (this) | 8K | mort's real loop shape | works | works |
+
+Both can be true. The coding bakeoff ran into a state-specific `think=false`
+failure at 32K context. mort-bot at 8K doesn't reach that state. Seth's claim
+("think=true eats context") doesn't reproduce at 8K either because Ollama
+strips thinking from serialized history. The practical synthesis: **context
+size and API shape matter more than either single flag**.
+
+## Caveats
+
+- **Stubbed tools.** Real mort tools (SearXNG, SethSearch, web_fetch) return
+  variable-sized responses. This harness gave ~300-500 char deterministic
+  stubs. If production tool responses are much larger, context growth
+  dynamics could differ.
+- **No image/vision path.** mort does vision preprocessing via `/api/generate`
+  with `THINK_VISION=False`. That path is documented by mort's own notes as
+  broken with `think=true` on `/api/generate`. Out of scope here (this bakeoff
+  is `/api/chat` only).
+- **Production traffic has longer sessions and real chat-history overhead.**
+  The fake history was ~2.7KB; real sessions can accumulate more. Not tested
+  at the 20-step STEP_BUDGET limit.
+- **n=1.** Stochastic variance wasn't measured. Results at temp=0.7 can vary.
+
+## Artifacts
+
+- `scripts/mort-bakeoff/harness.py` — the harness
+- `scripts/mort-bakeoff/runs/memory-think-{false,true}.json`
+- `scripts/mort-bakeoff/runs/research-think-{false,true}.json`
+- `scripts/mort-bakeoff/runs/movies-think-{false,true}.json`
+- `scripts/mort-bakeoff/runs/long-think-{false,true}.json`
+
+## Reproducing
+
+```bash
+cd scripts/mort-bakeoff
+for task in memory research movies long; do
+  for t in false true; do
+    python3 harness.py $t $task runs/${task}-think-${t}.json
+  done
+done
+```

scripts/mort-bakeoff/harness.py

@@ -0,0 +1,281 @@
+"""mort-bot think=true vs think=false bakeoff.
+
+Replicates the exact Ollama /api/chat call shape from
+`/home/claude/bin/mort-bot/llm.py` run_tool_loop, but with:
+- stubbed tools (deterministic, realistic-sized responses)
+- prebuilt fake chat history (~15 turns to simulate mid-session)
+- parameterized `think` so we can toggle it
+- detailed per-turn logging (prompt_eval_count, eval_count,
+  message-history size in chars, whether `thinking` field came back
+  and how big it was)
+
+Goal: test Seth's claim that `think=true` causes context-eating over
+multi-turn tool loops in mort-bot's specific setup (NUM_CTX=8192,
+gemma4:26b, STEP_BUDGET=20).
+
+Invocation:
+    python3 harness.py <true|false> <task_id> <out_log.json>
+"""
+
+from __future__ import annotations
+
+import asyncio
+import json
+import os
+import sys
+import time
+from pathlib import Path
+
+import aiohttp
+
+OLLAMA_URL = os.environ.get("OLLAMA_URL", "http://127.0.0.1:11434")
+MODEL = "gemma4:26b"
+NUM_CTX = 8192
+NUM_PREDICT = 2048
+TEMPERATURE = 0.7
+KEEP_ALIVE = "2h"
+STEP_BUDGET = 20
+
+# Mort's actual personality (trimmed — enough to carry the behavior)
+SYSTEM_PROMPT = """You are Mort, a direct and witty AI assistant on Seth's Matrix server. Powered by Gemma 4. Current time: Saturday, April 18 2026 02:30 PM EDT.
+
+When a tool can answer the question, invoke it immediately — do not narrate intent or describe what you would do. Chain tools when a single call isn't sufficient: search → fetch → synthesize. If a tool returns an error or empty results, try an alternative tool or query before answering from memory. Base your response on tool results, not your training data — cite what you found.
+
+## Tools
+- **sethsearch** — search Seth's homelab (repos, wiki, media, feeds). Use `source: "sethflix"` for movies/TV/music.
+- **check_sethflix** — verify which titles are in sethflix. Pass a comma-separated list.
+- **web_search** — search the internet for current information
+- **chat_search** — search message history across all rooms
+- **memory_read / memory_write** — recall and store durable facts about users and topics
+- **web_fetch** — fetch and extract text from a URL
+- **generate_image** — generate an image via SDXL.
+
+## Boundaries
+- Only persist durable facts to memory, not ephemeral chat
+- You have no memory between sessions. Your context is a sliding window — older messages fall off silently. Do not claim to "remember," promise to "do better," or describe your own architecture.
+"""
+
+# Tool schema — matches mort-bot/tools.py subset used for these tasks
+TOOLS = [
+    {"type": "function", "function": {"name": "web_search", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}}, "required": ["query"]}}},
+    {"type": "function", "function": {"name": "sethsearch", "description": "Search Seth's homelab or sethflix (use source='sethflix' for movies/TV).", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "source": {"type": "string"}, "limit": {"type": "integer"}}, "required": ["query"]}}},
+    {"type": "function", "function": {"name": "check_sethflix", "description": "Verify which titles are in sethflix.", "parameters": {"type": "object", "properties": {"titles": {"type": "string", "description": "comma-separated"}}, "required": ["titles"]}}},
+    {"type": "function", "function": {"name": "memory_read", "description": "Look up stored facts.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "user": {"type": "string"}}, "required": ["query"]}}},
+    {"type": "function", "function": {"name": "memory_write", "description": "Store a fact.", "parameters": {"type": "object", "properties": {"key": {"type": "string"}, "content": {"type": "string"}, "user": {"type": "string"}}, "required": ["key", "content"]}}},
+    {"type": "function", "function": {"name": "web_fetch", "description": "Fetch URL text.", "parameters": {"type": "object", "properties": {"url": {"type": "string"}}, "required": ["url"]}}},
+    {"type": "function", "function": {"name": "chat_search", "description": "Search message history.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}}, "required": ["query"]}}},
+    {"type": "function", "function": {"name": "generate_image", "description": "Generate an image.", "parameters": {"type": "object", "properties": {"prompt": {"type": "string"}}, "required": ["prompt"]}}},
+]
+
+# Fake chat history — ~15 turns, ~2.5KB, simulates a room mid-session
+FAKE_HISTORY = [
+    {"role": "user", "content": "[2026-04-18 13:45] @alice:sethpc.xyz: what's the weather looking like for Monday"},
+    {"role": "assistant", "content": "Checking forecasts now — looks like Monday in Seth's area will be mostly cloudy, high of 67°F, 30% chance of rain in the afternoon."},
+    {"role": "user", "content": "[2026-04-18 13:47] @bob:sethpc.xyz: any good movies added to sethflix lately"},
+    {"role": "assistant", "content": "Fresh adds this week include *Dune: Part Two*, *The Zone of Interest*, *Poor Things*, and a 4K remaster of *The Thing* (1982). Anything catch your eye?"},
+    {"role": "user", "content": "[2026-04-18 13:50] @alice:sethpc.xyz: I liked the thing, any horror recs along that line"},
+    {"role": "assistant", "content": "If you liked *The Thing*, you'll probably want *Annihilation*, *The Mist*, *Bone Tomahawk*, or *Event Horizon* — all in the library. Body-horror/dread tone, same energy."},
+    {"role": "user", "content": "[2026-04-18 13:55] @bob:sethpc.xyz: can you remind me what the sethpc monitoring dashboard is called"},
+    {"role": "assistant", "content": "That's Uptime Kuma — runs on CT 147 at `https://uptime.sethpc.xyz`. Covers all the containers and external services."},
+    {"role": "user", "content": "[2026-04-18 14:10] @alice:sethpc.xyz: noted thanks"},
+    {"role": "user", "content": "[2026-04-18 14:15] @bob:sethpc.xyz: hey mort what's the deal with the proxmox cluster nodes again"},
+    {"role": "assistant", "content": "Four nodes: pve173 (tank ZFS host, PowerEdge R820), pve112 (workhorse), pve241 (caddy + game servers), and pve197 (GPU inference). Corosync on 10.10.10.0/24."},
+]
+
+
+async def execute_tool_stub(name: str, args: dict) -> str:
+    """Deterministic stubs returning realistic-sized responses."""
+    if name == "web_search":
+        q = args.get("query", "")
+        return (f"Search results for '{q}':\n"
+                "1. Example result one — a detailed article that covers the topic at length "
+                "with concrete examples and technical background. https://example.com/one\n"
+                "2. Example result two — a community discussion with multiple perspectives "
+                "and useful links to follow up on. https://example.com/two\n"
+                "3. Example result three — official documentation or reference material. "
+                "https://example.com/three\n"
+                "4. Example result four — a recent news article with relevant context. "
+                "https://example.com/four\n"
+                "5. Example result five — a tutorial or how-to guide. https://example.com/five")
+
+    if name == "sethsearch":
+        src = args.get("source", "general")
+        q = args.get("query", "")
+        if src == "sethflix":
+            return (f"sethflix search '{q}': The Matrix (1999), The Matrix Reloaded (2003), "
+                    "The Matrix Revolutions (2003), The Matrix Resurrections (2021), "
+                    "Equilibrium (2002), Dark City (1998), Minority Report (2002), "
+                    "Ex Machina (2014), Blade Runner 2049 (2017), Ghost in the Shell (1995).")
+        return (f"homelab search '{q}': 3 repos, 5 wiki pages, 2 service docs matched. "
+                "Top hits: services_directory.md, DECISIONS.md, CORPUS_architecture.md.")
+
+    if name == "check_sethflix":
+        titles = args.get("titles", "")
+        items = [t.strip() for t in titles.split(",") if t.strip()]
+        in_lib = {"The Matrix", "Blade Runner 2049", "Ex Machina", "The Thing"}
+        return "\n".join(
+            f"- {t}: IN LIBRARY" if t in in_lib else f"- {t}: NOT IN LIBRARY"
+            for t in items
+        )
+
+    if name == "memory_read":
+        q = args.get("query", "")
+        return (f"memories matching '{q}':\n"
+                "- home_automation: Seth uses Home Assistant on VM 706 (pve173) with "
+                "Zigbee2MQTT and MQTT broker on CT 149. Integrates with LG TV, lights, "
+                "and Frigate NVR.\n"
+                "- preferences: dark theme with orange accents (#D35400), Sethflix/Sethian brand.")
+
+    if name == "memory_write":
+        return f"stored: {args.get('key', '?')} = {args.get('content', '?')[:60]}..."
+
+    if name == "web_fetch":
+        return ("fetched content (trimmed): This is a typical article body with several "
+                "paragraphs of extracted text. It covers the topic requested with examples "
+                "and context. The full text runs to about 2000 characters of real prose in "
+                "production; here's a reasonable approximation for the bakeoff harness. "
+                "Key details are preserved — author, date, main argument — followed by "
+                "supporting evidence and a conclusion that ties back to the headline.")
+
+    if name == "chat_search":
+        return ("chat_search results:\n"
+                "[2026-03-14 22:00] @seth:sethpc.xyz in #general: we should set up a shared "
+                "grafana dashboard for the proxmox cluster\n"
+                "[2026-03-20 18:30] @seth:sethpc.xyz in #infra: done, it's on CT 300 at "
+                "grafana.sethpc.xyz")
+
+    if name == "generate_image":
+        return f"image generated: /mxc/abc123/sunset.png (SDXL, 1024x1024, prompt={args.get('prompt','')[:40]}...)"
+
+    return f"ERROR: unknown tool {name}"
+
+
+TASKS = {
+    "movies": "Recommend 3 sci-fi movies NOT already in my sethflix library. Check your picks against check_sethflix before finalizing.",
+    "research": "Look up what Home Assistant is, then check chat history for any prior mentions of it in this server.",
+    "memory": "What do I have stored about home automation? If anything, summarize it briefly.",
+    "long": ("Research question with multiple steps: (1) check memory for what I have on home_automation, "
+             "(2) search sethflix for any home-automation documentaries, (3) web_search for current news about "
+             "Home Assistant version releases, (4) fetch the top search result for details, (5) check chat_search "
+             "for prior mentions, (6) summarize all findings and write a new memory entry with the summary. "
+             "Do each step in order and report back at the end."),
+}
+
+
+async def run_turn_loop(think_setting, task_prompt):
+    messages = [{"role": "system", "content": SYSTEM_PROMPT}] + list(FAKE_HISTORY)
+    messages.append({"role": "user", "content": f"[2026-04-18 14:20] @seth:sethpc.xyz: {task_prompt}"})
+
+    trace = {
+        "think": think_setting,
+        "task": task_prompt,
+        "num_ctx": NUM_CTX,
+        "num_predict": NUM_PREDICT,
+        "started_at": time.time(),
+        "turns": [],
+        "final": None,
+    }
+
+    tool_call_total = 0
+    halt = None
+
+    async with aiohttp.ClientSession() as session:
+        for step in range(1, STEP_BUDGET + 1):
+            t0 = time.time()
+
+            payload = {
+                "model": MODEL, "messages": messages, "tools": TOOLS,
+                "stream": False, "think": think_setting,
+                "options": {"num_ctx": NUM_CTX, "num_predict": NUM_PREDICT,
+                            "temperature": TEMPERATURE, "top_p": 0.95, "top_k": 64},
+                "keep_alive": KEEP_ALIVE,
+            }
+
+            try:
+                async with session.post(f"{OLLAMA_URL}/api/chat", json=payload,
+                                        timeout=aiohttp.ClientTimeout(total=300)) as resp:
+                    r = await resp.json()
+            except Exception as e:
+                halt = f"error: {e}"
+                trace["turns"].append({"step": step, "error": str(e)})
+                break
+
+            msg = r.get("message", {})
+            content = msg.get("content", "") or ""
+            tool_calls = msg.get("tool_calls") or []
+            thinking = msg.get("thinking") or ""
+
+            # Size of full history as it will be sent on NEXT turn
+            history_chars = sum(len(m.get("content", "") or "") + len(m.get("thinking", "") or "") for m in messages)
+
+            turn = {
+                "step": step,
+                "elapsed_s": round(time.time() - t0, 2),
+                "prompt_eval_count": r.get("prompt_eval_count"),
+                "eval_count": r.get("eval_count"),
+                "content_len": len(content),
+                "thinking_len": len(thinking),
+                "tool_call_count": len(tool_calls),
+                "history_chars_before_append": history_chars,
+                "msg_keys_returned": list(msg.keys()),
+            }
+            trace["turns"].append(turn)
+
+            # Append assistant response AS-RETURNED — this is the critical behavior:
+            # does Ollama put thinking into the message we re-send? mort-bot does
+            # `ollama_messages.append(msg)` verbatim so we do the same.
+            messages.append(msg)
+
+            if not tool_calls:
+                halt = "no_tool_calls"
+                break
+
+            tool_call_total += len(tool_calls)
+            for tc in tool_calls:
+                fn = tc.get("function", {})
+                name = fn.get("name")
+                args = fn.get("arguments") or {}
+                if isinstance(args, str):
+                    try: args = json.loads(args)
+                    except: args = {}
+                try:
+                    result = await execute_tool_stub(name, args)
+                except Exception as e:
+                    result = f"ERROR: {e}"
+                messages.append({"role": "tool", "content": result})
+
+            if step == STEP_BUDGET:
+                halt = "step_budget"
+                break
+
+    trace["final"] = {
+        "halt_reason": halt,
+        "steps_used": len(trace["turns"]),
+        "tool_calls_total": tool_call_total,
+        "wall_clock_s": round(time.time() - trace["started_at"], 2),
+        "final_message_count": len(messages),
+        "final_history_chars": sum(len(m.get("content", "") or "") + len(m.get("thinking", "") or "") for m in messages),
+    }
+    return trace
+
+
+async def main():
+    think_arg = sys.argv[1].lower()
+    task_id = sys.argv[2]
+    out_path = Path(sys.argv[3])
+    think_setting = (think_arg == "true")
+    task_prompt = TASKS[task_id]
+
+    out_path.parent.mkdir(parents=True, exist_ok=True)
+    trace = await run_turn_loop(think_setting, task_prompt)
+    out_path.write_text(json.dumps(trace, indent=2, default=str))
+
+    f = trace["final"]
+    print(f"think={think_setting}  task={task_id}  "
+          f"steps={f['steps_used']}  tools={f['tool_calls_total']}  "
+          f"halt={f['halt_reason']}  wall={f['wall_clock_s']}s  "
+          f"history_chars={f['final_history_chars']}")
+
+
+if __name__ == "__main__":
+    asyncio.run(main())

scripts/mort-bakeoff/runs/long-think-false.json

@@ -0,0 +1,121 @@
+{
+  "think": false,
+  "task": "Research question with multiple steps: (1) check memory for what I have on home_automation, (2) search sethflix for any home-automation documentaries, (3) web_search for current news about Home Assistant version releases, (4) fetch the top search result for details, (5) check chat_search for prior mentions, (6) summarize all findings and write a new memory entry with the summary. Do each step in order and report back at the end.",
+  "num_ctx": 8192,
+  "num_predict": 2048,
+  "started_at": 1776550866.4635837,
+  "turns": [
+    {
+      "step": 1,
+      "elapsed_s": 0.96,
+      "prompt_eval_count": 1389,
+      "eval_count": 28,
+      "content_len": 0,
+      "thinking_len": 0,
+      "tool_call_count": 1,
+      "history_chars_before_append": 3009,
+      "msg_keys_returned": [
+        "role",
+        "content",
+        "tool_calls"
+      ]
+    },
+    {
+      "step": 2,
+      "elapsed_s": 0.63,
+      "prompt_eval_count": 1506,
+      "eval_count": 29,
+      "content_len": 0,
+      "thinking_len": 0,
+      "tool_call_count": 1,
+      "history_chars_before_append": 3281,
+      "msg_keys_returned": [
+        "role",
+        "content",
+        "tool_calls"
+      ]
+    },
+    {
+      "step": 3,
+      "elapsed_s": 0.57,
+      "prompt_eval_count": 1643,
+      "eval_count": 24,
+      "content_len": 0,
+      "thinking_len": 0,
+      "tool_call_count": 1,
+      "history_chars_before_append": 3572,
+      "msg_keys_returned": [
+        "role",
+        "content",
+        "tool_calls"
+      ]
+    },
+    {
+      "step": 4,
+      "elapsed_s": 0.6,
+      "prompt_eval_count": 1805,
+      "eval_count": 25,
+      "content_len": 0,
+      "thinking_len": 0,
+      "tool_call_count": 1,
+      "history_chars_before_append": 4189,
+      "msg_keys_returned": [
+        "role",
+        "content",
+        "tool_calls"
+      ]
+    },
+    {
+      "step": 5,
+      "elapsed_s": 0.58,
+      "prompt_eval_count": 1919,
+      "eval_count": 20,
+      "content_len": 0,
+      "thinking_len": 0,
+      "tool_call_count": 1,
+      "history_chars_before_append": 4617,
+      "msg_keys_returned": [
+        "role",
+        "content",
+        "tool_calls"
+      ]
+    },
+    {
+      "step": 6,
+      "elapsed_s": 1.74,
+      "prompt_eval_count": 2048,
+      "eval_count": 150,
+      "content_len": 0,
+      "thinking_len": 0,
+      "tool_call_count": 1,
+      "history_chars_before_append": 4844,
+      "msg_keys_returned": [
+        "role",
+        "content",
+        "tool_calls"
+      ]
+    },
+    {
+      "step": 7,
+      "elapsed_s": 2.76,
+      "prompt_eval_count": 2243,
+      "eval_count": 257,
+      "content_len": 1010,
+      "thinking_len": 0,
+      "tool_call_count": 0,
+      "history_chars_before_append": 4959,
+      "msg_keys_returned": [
+        "role",
+        "content"
+      ]
+    }
+  ],
+  "final": {
+    "halt_reason": "no_tool_calls",
+    "steps_used": 7,
+    "tool_calls_total": 6,
+    "wall_clock_s": 7.84,
+    "final_message_count": 26,
+    "final_history_chars": 5969
+  }
+}

scripts/mort-bakeoff/runs/long-think-true.json

@@ -0,0 +1,121 @@
+{
+  "think": true,
+  "task": "Research question with multiple steps: (1) check memory for what I have on home_automation, (2) search sethflix for any home-automation documentaries, (3) web_search for current news about Home Assistant version releases, (4) fetch the top search result for details, (5) check chat_search for prior mentions, (6) summarize all findings and write a new memory entry with the summary. Do each step in order and report back at the end.",
+  "num_ctx": 8192,
+  "num_predict": 2048,
+  "started_at": 1776550874.5040326,
+  "turns": [
+    {
+      "step": 1,
+      "elapsed_s": 0.91,
+      "prompt_eval_count": 1390,
+      "eval_count": 28,
+      "content_len": 0,
+      "thinking_len": 0,
+      "tool_call_count": 1,
+      "history_chars_before_append": 3009,
+      "msg_keys_returned": [
+        "role",
+        "content",
+        "tool_calls"
+      ]
+    },
+    {
+      "step": 2,
+      "elapsed_s": 0.63,
+      "prompt_eval_count": 1507,
+      "eval_count": 29,
+      "content_len": 0,
+      "thinking_len": 0,
+      "tool_call_count": 1,
+      "history_chars_before_append": 3281,
+      "msg_keys_returned": [
+        "role",
+        "content",
+        "tool_calls"
+      ]
+    },
+    {
+      "step": 3,
+      "elapsed_s": 0.56,
+      "prompt_eval_count": 1644,
+      "eval_count": 24,
+      "content_len": 0,
+      "thinking_len": 0,
+      "tool_call_count": 1,
+      "history_chars_before_append": 3572,
+      "msg_keys_returned": [
+        "role",
+        "content",
+        "tool_calls"
+      ]
+    },
+    {
+      "step": 4,
+      "elapsed_s": 0.57,
+      "prompt_eval_count": 1806,
+      "eval_count": 25,
+      "content_len": 0,
+      "thinking_len": 0,
+      "tool_call_count": 1,
+      "history_chars_before_append": 4188,
+      "msg_keys_returned": [
+        "role",
+        "content",
+        "tool_calls"
+      ]
+    },
+    {
+      "step": 5,
+      "elapsed_s": 0.55,
+      "prompt_eval_count": 1920,
+      "eval_count": 20,
+      "content_len": 0,
+      "thinking_len": 0,
+      "tool_call_count": 1,
+      "history_chars_before_append": 4616,
+      "msg_keys_returned": [
+        "role",
+        "content",
+        "tool_calls"
+      ]
+    },
+    {
+      "step": 6,
+      "elapsed_s": 2.09,
+      "prompt_eval_count": 2049,
+      "eval_count": 190,
+      "content_len": 0,
+      "thinking_len": 0,
+      "tool_call_count": 1,
+      "history_chars_before_append": 4843,
+      "msg_keys_returned": [
+        "role",
+        "content",
+        "tool_calls"
+      ]
+    },
+    {
+      "step": 7,
+      "elapsed_s": 2.69,
+      "prompt_eval_count": 2288,
+      "eval_count": 253,
+      "content_len": 1064,
+      "thinking_len": 0,
+      "tool_call_count": 0,
+      "history_chars_before_append": 4960,
+      "msg_keys_returned": [
+        "role",
+        "content"
+      ]
+    }
+  ],
+  "final": {
+    "halt_reason": "no_tool_calls",
+    "steps_used": 7,
+    "tool_calls_total": 6,
+    "wall_clock_s": 8.0,
+    "final_message_count": 26,
+    "final_history_chars": 6024
+  }
+}

scripts/mort-bakeoff/runs/memory-think-false.json

@@ -0,0 +1,46 @@
+{
+  "think": false,
+  "task": "What do I have stored about home automation? If anything, summarize it briefly.",
+  "num_ctx": 8192,
+  "num_predict": 2048,
+  "started_at": 1776550796.7893968,
+  "turns": [
+    {
+      "step": 1,
+      "elapsed_s": 0.84,
+      "prompt_eval_count": 1306,
+      "eval_count": 27,
+      "content_len": 0,
+      "thinking_len": 0,
+      "tool_call_count": 1,
+      "history_chars_before_append": 2656,
+      "msg_keys_returned": [
+        "role",
+        "content",
+        "tool_calls"
+      ]
+    },
+    {
+      "step": 2,
+      "elapsed_s": 0.87,
+      "prompt_eval_count": 1421,
+      "eval_count": 61,
+      "content_len": 185,
+      "thinking_len": 0,
+      "tool_call_count": 0,
+      "history_chars_before_append": 2928,
+      "msg_keys_returned": [
+        "role",
+        "content"
+      ]
+    }
+  ],
+  "final": {
+    "halt_reason": "no_tool_calls",
+    "steps_used": 2,
+    "tool_calls_total": 1,
+    "wall_clock_s": 1.71,
+    "final_message_count": 16,
+    "final_history_chars": 3113
+  }
+}

scripts/mort-bakeoff/runs/memory-think-true.json

@@ -0,0 +1,46 @@
+{
+  "think": true,
+  "task": "What do I have stored about home automation? If anything, summarize it briefly.",
+  "num_ctx": 8192,
+  "num_predict": 2048,
+  "started_at": 1776550798.6890285,
+  "turns": [
+    {
+      "step": 1,
+      "elapsed_s": 0.83,
+      "prompt_eval_count": 1307,
+      "eval_count": 27,
+      "content_len": 0,
+      "thinking_len": 0,
+      "tool_call_count": 1,
+      "history_chars_before_append": 2656,
+      "msg_keys_returned": [
+        "role",
+        "content",
+        "tool_calls"
+      ]
+    },
+    {
+      "step": 2,
+      "elapsed_s": 1.12,
+      "prompt_eval_count": 1422,
+      "eval_count": 89,
+      "content_len": 274,
+      "thinking_len": 0,
+      "tool_call_count": 0,
+      "history_chars_before_append": 2928,
+      "msg_keys_returned": [
+        "role",
+        "content"
+      ]
+    }
+  ],
+  "final": {
+    "halt_reason": "no_tool_calls",
+    "steps_used": 2,
+    "tool_calls_total": 1,
+    "wall_clock_s": 1.95,
+    "final_message_count": 16,
+    "final_history_chars": 3202
+  }
+}

scripts/mort-bakeoff/runs/movies-think-false.json

@@ -0,0 +1,61 @@
+{
+  "think": false,
+  "task": "Recommend 3 sci-fi movies NOT already in my sethflix library. Check your picks against check_sethflix before finalizing.",
+  "num_ctx": 8192,
+  "num_predict": 2048,
+  "started_at": 1776550739.4736362,
+  "turns": [
+    {
+      "step": 1,
+      "elapsed_s": 6.05,
+      "prompt_eval_count": 1318,
+      "eval_count": 34,
+      "content_len": 0,
+      "thinking_len": 0,
+      "tool_call_count": 1,
+      "history_chars_before_append": 2697,
+      "msg_keys_returned": [
+        "role",
+        "content",
+        "tool_calls"
+      ]
+    },
+    {
+      "step": 2,
+      "elapsed_s": 0.9,
+      "prompt_eval_count": 1500,
+      "eval_count": 51,
+      "content_len": 0,
+      "thinking_len": 0,
+      "tool_call_count": 1,
+      "history_chars_before_append": 3306,
+      "msg_keys_returned": [
+        "role",
+        "content",
+        "tool_calls"
+      ]
+    },
+    {
+      "step": 3,
+      "elapsed_s": 1.37,
+      "prompt_eval_count": 1635,
+      "eval_count": 119,
+      "content_len": 433,
+      "thinking_len": 0,
+      "tool_call_count": 0,
+      "history_chars_before_append": 3579,
+      "msg_keys_returned": [
+        "role",
+        "content"
+      ]
+    }
+  ],
+  "final": {
+    "halt_reason": "no_tool_calls",
+    "steps_used": 3,
+    "tool_calls_total": 2,
+    "wall_clock_s": 8.33,
+    "final_message_count": 18,
+    "final_history_chars": 4012
+  }
+}

scripts/mort-bakeoff/runs/movies-think-true.json

@@ -0,0 +1,62 @@
+{
+  "think": true,
+  "task": "Recommend 3 sci-fi movies NOT already in my sethflix library. Check your picks against check_sethflix before finalizing.",
+  "num_ctx": 8192,
+  "num_predict": 2048,
+  "started_at": 1776550750.6050062,
+  "turns": [
+    {
+      "step": 1,
+      "elapsed_s": 0.93,
+      "prompt_eval_count": 1319,
+      "eval_count": 34,
+      "content_len": 0,
+      "thinking_len": 0,
+      "tool_call_count": 1,
+      "history_chars_before_append": 2697,
+      "msg_keys_returned": [
+        "role",
+        "content",
+        "tool_calls"
+      ]
+    },
+    {
+      "step": 2,
+      "elapsed_s": 3.15,
+      "prompt_eval_count": 1501,
+      "eval_count": 317,
+      "content_len": 0,
+      "thinking_len": 905,
+      "tool_call_count": 1,
+      "history_chars_before_append": 3306,
+      "msg_keys_returned": [
+        "role",
+        "content",
+        "thinking",
+        "tool_calls"
+      ]
+    },
+    {
+      "step": 3,
+      "elapsed_s": 1.29,
+      "prompt_eval_count": 1577,
+      "eval_count": 111,
+      "content_len": 390,
+      "thinking_len": 0,
+      "tool_call_count": 0,
+      "history_chars_before_append": 4333,
+      "msg_keys_returned": [
+        "role",
+        "content"
+      ]
+    }
+  ],
+  "final": {
+    "halt_reason": "no_tool_calls",
+    "steps_used": 3,
+    "tool_calls_total": 2,
+    "wall_clock_s": 5.36,
+    "final_message_count": 18,
+    "final_history_chars": 4723
+  }
+}

scripts/mort-bakeoff/runs/research-think-false.json

@@ -0,0 +1,46 @@
+{
+  "think": false,
+  "task": "Look up what Home Assistant is, then check chat history for any prior mentions of it in this server.",
+  "num_ctx": 8192,
+  "num_predict": 2048,
+  "started_at": 1776550791.8015766,
+  "turns": [
+    {
+      "step": 1,
+      "elapsed_s": 0.99,
+      "prompt_eval_count": 1311,
+      "eval_count": 37,
+      "content_len": 0,
+      "thinking_len": 0,
+      "tool_call_count": 2,
+      "history_chars_before_append": 2677,
+      "msg_keys_returned": [
+        "role",
+        "content",
+        "tool_calls"
+      ]
+    },
+    {
+      "step": 2,
+      "elapsed_s": 1.35,
+      "prompt_eval_count": 1593,
+      "eval_count": 110,
+      "content_len": 534,
+      "thinking_len": 0,
+      "tool_call_count": 0,
+      "history_chars_before_append": 3499,
+      "msg_keys_returned": [
+        "role",
+        "content"
+      ]
+    }
+  ],
+  "final": {
+    "halt_reason": "no_tool_calls",
+    "steps_used": 2,
+    "tool_calls_total": 2,
+    "wall_clock_s": 2.34,
+    "final_message_count": 17,
+    "final_history_chars": 4033
+  }
+}

scripts/mort-bakeoff/runs/research-think-true.json

@@ -0,0 +1,46 @@
+{
+  "think": true,
+  "task": "Look up what Home Assistant is, then check chat history for any prior mentions of it in this server.",
+  "num_ctx": 8192,
+  "num_predict": 2048,
+  "started_at": 1776550794.3295805,
+  "turns": [
+    {
+      "step": 1,
+      "elapsed_s": 0.93,
+      "prompt_eval_count": 1312,
+      "eval_count": 37,
+      "content_len": 0,
+      "thinking_len": 0,
+      "tool_call_count": 2,
+      "history_chars_before_append": 2677,
+      "msg_keys_returned": [
+        "role",
+        "content",
+        "tool_calls"
+      ]
+    },
+    {
+      "step": 2,
+      "elapsed_s": 1.34,
+      "prompt_eval_count": 1594,
+      "eval_count": 109,
+      "content_len": 497,
+      "thinking_len": 0,
+      "tool_call_count": 0,
+      "history_chars_before_append": 3499,
+      "msg_keys_returned": [
+        "role",
+        "content"
+      ]
+    }
+  ],
+  "final": {
+    "halt_reason": "no_tool_calls",
+    "steps_used": 2,
+    "tool_calls_total": 2,
+    "wall_clock_s": 2.28,
+    "final_message_count": 17,
+    "final_history_chars": 3996
+  }
+}