From 8436a91571984d26993062a3a1c8f3028c8bb590 Mon Sep 17 00:00:00 2001 From: Mortdecai Date: Sat, 18 Apr 2026 18:23:43 -0400 Subject: [PATCH] feat: mort-bot think=true vs think=false bakeoff MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Seth's challenge: "we experienced this context eating with every implementation that had think=true. mort-bot runs a loop. Can you do a bake-off?" Built a harness that replicates mort-bot's /api/chat loop verbatim (num_ctx=8192, num_predict=2048, temperature=0.7, gemma4:26b, STEP_BUDGET=20, exact payload shape) but with stubbed tools and a prebuilt 15-turn fake chat history. Ran 4 tasks × 2 think settings. Finding: on Ollama 0.20.4 the "thinking eats context" concern does NOT reproduce. Direct evidence: - Movies task step 2 (think=true) returned 905 chars of thinking. - Step 3 prompt_eval_count delta: +76 tokens (think=true) vs +135 tokens (think=false). If thinking had accumulated in the prompt, think=true would have grown by +360 tokens, not shrunk. - Ollama's chat template strips the `thinking` field when serializing assistant turns for subsequent prompts. All 4 tasks × 2 settings produced identical step counts and tool counts. Wall clocks comparable. Gemma only actually generated thinking on 1 of 4 tasks (the one with check_sethflix verify-loop); on the others with think=true it emitted 0 thinking tokens. Reconciled with the earlier coding-agent bakeoff: the two findings are orthogonal. Coding bakeoff was at num_ctx=32K with a different harness; mort at 8K doesn't touch the silent-stop regime either way. Seth's prior may have been correct on an older Ollama or in a different API shape (/api/generate has its own issues) but does not reproduce here. Concrete recommendation: mort-bot THINK=False is defensible but not load-bearing; THINK=True or unset-default would also work. Keep as-is unless a different need arises. New: docs/reference/mort-bakeoff-2026-04-18.md, scripts/mort-bakeoff/ (harness + 8 run logs). README updated with pointer. --- .secrets.baseline | 2 +- README.md | 3 +- docs/reference/mort-bakeoff-2026-04-18.md | 155 ++++++++++ scripts/mort-bakeoff/harness.py | 281 ++++++++++++++++++ .../mort-bakeoff/runs/long-think-false.json | 121 ++++++++ .../mort-bakeoff/runs/long-think-true.json | 121 ++++++++ .../mort-bakeoff/runs/memory-think-false.json | 46 +++ .../mort-bakeoff/runs/memory-think-true.json | 46 +++ .../mort-bakeoff/runs/movies-think-false.json | 61 ++++ .../mort-bakeoff/runs/movies-think-true.json | 62 ++++ .../runs/research-think-false.json | 46 +++ .../runs/research-think-true.json | 46 +++ 12 files changed, 988 insertions(+), 2 deletions(-) create mode 100644 docs/reference/mort-bakeoff-2026-04-18.md create mode 100644 scripts/mort-bakeoff/harness.py create mode 100644 scripts/mort-bakeoff/runs/long-think-false.json create mode 100644 scripts/mort-bakeoff/runs/long-think-true.json create mode 100644 scripts/mort-bakeoff/runs/memory-think-false.json create mode 100644 scripts/mort-bakeoff/runs/memory-think-true.json create mode 100644 scripts/mort-bakeoff/runs/movies-think-false.json create mode 100644 scripts/mort-bakeoff/runs/movies-think-true.json create mode 100644 scripts/mort-bakeoff/runs/research-think-false.json create mode 100644 scripts/mort-bakeoff/runs/research-think-true.json diff --git a/.secrets.baseline b/.secrets.baseline index 83edc92..f690597 100644 --- a/.secrets.baseline +++ b/.secrets.baseline @@ -14579,5 +14579,5 @@ } ] }, - "generated_at": "2026-04-18T22:14:05Z" + "generated_at": "2026-04-18T22:23:43Z" } diff --git a/README.md b/README.md index 2d9ab02..659fbad 100644 --- a/README.md +++ b/README.md @@ -15,7 +15,8 @@ Research corpus and implementation guidance for Google Gemma 4, based on product | `CORPUS_benchmarks.md` | Full benchmark table vs Gemma 3, arena scores, agentic scores | When comparing Gemma 4 to alternatives | | `CORPUS_tool_calling_format.md` | Native token format + JSON API format for function calling | When implementing tool calling | | `CORPUS_cli_coding_agent.md` | Positioning Gemma 4 for CLI coding agent use (openclaw / open code / pi / hermes / aider style). Honest take on what Google did and didn't measure, head-to-head with `qwen3-coder:30b`, homelab setup pointer | When scoping a CLI coding agent or deciding Gemma 4 vs Qwen3-Coder | -| `docs/reference/bakeoff-2026-04-18.md` | Raw results: CLI-coding-agent bakeoff of gemma4:26b / gemma4:31b / qwen3-coder:30b on steel141 3090 Ti. **31B clean, Qwen3-Coder correct but chatty, 26B reproducibly silent-stops at write_file.** Harness at `scripts/bakeoff/` | When deciding which model to back a CLI agent with, or debugging a similar tool-call halt | +| `docs/reference/bakeoff-2026-04-18.md` | CLI-coding-agent bakeoff on 3090 Ti. **Rounds 1/2 misidentified the cause; Round 3 (the correct one): `think: false` silent-stops gemma4:26b at certain multi-turn states on 32K context.** 31B and Qwen3-Coder robust to the flag. Harness at `scripts/bakeoff/` | When deciding which model to back a CLI agent with, writing a custom agent payload, or debugging a silent tool-call halt | +| `docs/reference/mort-bakeoff-2026-04-18.md` | mort-bot-specific `think=true` vs `think=false` bakeoff on mort's actual loop shape (gemma4:26b, num_ctx=8192). **Thinking does NOT accumulate in context on Ollama 0.20.4** — strips it from serialized history. Both settings behave identically on step counts, tool counts, wall clock. Harness at `scripts/mort-bakeoff/` | When deciding mort-bot's THINK env var, or when someone claims "think=true eats context" without pinning an Ollama version | | `tooling/` | **Canonical upstream tooling** — real scripts, notebooks, model cards, and configs pulled from Google / HF / framework maintainers (147 files). Subdirs: `google-official/`, `huggingface/`, `inference-frameworks/`, `gemma-family/`, `fine-tuning/`. See `tooling/README.md` for index and findings that update the older `CORPUS_*` docs | When you need authoritative source material — model cards, chat templates, fine-tuning recipes, serving commands for vLLM / llama.cpp / MLX, or to scope a specialized sibling (ShieldGemma, EmbeddingGemma, etc.) | ## Source Projects diff --git a/docs/reference/mort-bakeoff-2026-04-18.md b/docs/reference/mort-bakeoff-2026-04-18.md new file mode 100644 index 0000000..a9114bd --- /dev/null +++ b/docs/reference/mort-bakeoff-2026-04-18.md @@ -0,0 +1,155 @@ +# mort-bot `think=true` vs `think=false` Bakeoff — 2026-04-18 + +> Follow-up to Seth's challenge: "we experienced this context eating with every +> implementation that had think=true. mort-bot runs a loop. Can you do a +> bake-off to see if that bot would actually perform better with thinking on?" +> +> Short answer: **no measurable difference on Ollama 0.20.4**. The +> "thinking-eats-context" concern doesn't reproduce in mort-bot's current +> loop shape because Ollama's chat template strips the `thinking` field from +> serialized history when it builds the prompt for subsequent turns. Either +> setting is defensible. + +## Setup + +- Harness: `scripts/mort-bakeoff/harness.py` — replicates mort-bot `llm.py` + `run_tool_loop` call shape verbatim (model, options, payload structure, + `messages.append(msg)` behavior), but with stubbed tools and a prebuilt + ~15-turn fake chat history to simulate mid-session state. +- Host / Ollama: steel141, 3090 Ti, Ollama 0.20.4 +- Exact config match to mort-bot production: `gemma4:26b`, `num_ctx=8192`, + `num_predict=2048`, `temperature=0.7`, `top_p=0.95`, `top_k=64`, + `keep_alive=2h`, `STEP_BUDGET=20` +- Tasks: 4 scenarios representative of real traffic (movies, research, + memory, long-chain research) +- n=1 per (task, think) cell — bakeoff, not benchmark + +## Results + +| Task | Think | Steps | Tools | Peak prompt | Thinking generated | Wall | +|---|---|---|---|---|---|---| +| memory | false | 2 | 1 | 1421 | 0 tok | 1.7s | +| memory | true | 2 | 1 | 1422 | 0 tok | 2.0s | +| research | false | 2 | 2 | 1593 | 0 tok | 2.3s | +| research | true | 2 | 2 | 1594 | 0 tok | 2.3s | +| movies | false | 3 | 2 | 1635 | 0 tok | 8.3s | +| movies | true | 3 | 2 | 1577 | 905 chars / ~226 tok | 5.4s | +| long | false | 7 | 6 | 2243 | 0 tok | 7.8s | +| long | true | 7 | 6 | 2288 | 0 tok | 8.0s | + +Every (task, think) pair produced **identical step counts and tool counts**. + +## Does thinking accumulate in context? + +**No — verified directly.** + +On the `movies` task, step 2 with `think=true` returned 905 chars (~226 tok) +in a separate `thinking` field. My harness then appends the full message dict +(including `thinking`) to `messages` before the next request — exactly what +mort-bot's `ollama_messages.append(msg)` does. + +Step 2 → Step 3 prompt_eval_count delta: + +| Setting | Delta | What that means | +|---|---|---| +| think=false | +135 tok | Tool result only | +| think=true | +76 tok | Tool result only — thinking was **stripped** | + +If thinking had accumulated, the think=true delta would have been ~360 tokens +(tool result + thinking). Instead it was smaller than think=false's delta. +**Ollama 0.20.4's chat template does not include the `thinking` field when +serializing an assistant turn for subsequent prompts.** Thinking is a per-turn +response annotation, not a persisted conversation channel. + +## Does thinking actually happen? + +Gemma 4 is conservative. With `think=true`, it chose to generate thinking +tokens on **1 of 4 tasks**: + +- `memory` — 0 thinking tokens (simple lookup) +- `research` — 0 thinking tokens (clear sequential plan) +- `long` — 0 thinking tokens across 7 steps (explicit step list to follow) +- `movies` — 905 chars thinking on step 2 (the only task requiring + verification logic: check candidates → filter IN LIBRARY → replace → re-check) + +The model appears to decide whether to think based on whether the task has +uncertainty to reason about. Following explicit multi-step instructions +doesn't trigger it; generating + filtering recommendations does. + +## Answer to Seth's claim + +Seth's prior: "we experienced this context eating with every implementation +that had think=true." + +On current Ollama (0.20.4) with mort-bot's loop shape: **the claim does not +reproduce.** Possibilities for why Seth saw it before: + +1. **Older Ollama version.** Earlier 0.x releases may have serialized `thinking` + into subsequent prompts. Not tested here. +2. **Different API shape.** `/api/generate` behaves differently from `/api/chat` + — mort-bot's own CLAUDE.md notes `/api/generate` with `think=true` returns + empty responses. That's a separate, real bug, not a context-growth bug. +3. **AI_Visualizer-shaped pipelines** generate into `content` where thinking + tokens explicitly eat the `num_predict` budget. That is a real failure mode + and the original "always think:false" guidance addresses it correctly. +4. **Confounded by other issues** — context truncation, model quirks, silent + failures — misattributed to thinking. + +The production `THINK=False` setting was defensible when adopted and remains +defensible now. It's just not load-bearing in the way the original guidance +suggested. + +## Concrete recommendation for mort-bot + +1. **Keep `THINK=False` as-is, or try unset-default, or `THINK=True` — pick based on whether you want the slight quality hedge on reasoning-heavy turns (like the movies check_sethflix verify-loop). No context-growth penalty either way on 0.20.4.** +2. **Don't backport the "CLI coding agent" finding from `docs/reference/bakeoff-2026-04-18.md`** — that one was at `num_ctx=32768` with a coding harness, different regime. Mort's `num_ctx=8192` doesn't touch the silent-stop trigger. +3. **If Ollama versions drift**, re-run this harness. The stripping behavior is an Ollama implementation detail; a future version could change it. + +## Reconciling with the earlier coding-agent bakeoff + +The two findings are orthogonal: + +| Bakeoff | Context | Harness | think=false | think=true | +|---|---|---|---|---| +| CLI coding (Round 3) | 32K | custom coding loop | 26B silent-stops | works | +| mort-bot (this) | 8K | mort's real loop shape | works | works | + +Both can be true. The coding bakeoff ran into a state-specific `think=false` +failure at 32K context. mort-bot at 8K doesn't reach that state. Seth's claim +("think=true eats context") doesn't reproduce at 8K either because Ollama +strips thinking from serialized history. The practical synthesis: **context +size and API shape matter more than either single flag**. + +## Caveats + +- **Stubbed tools.** Real mort tools (SearXNG, SethSearch, web_fetch) return + variable-sized responses. This harness gave ~300-500 char deterministic + stubs. If production tool responses are much larger, context growth + dynamics could differ. +- **No image/vision path.** mort does vision preprocessing via `/api/generate` + with `THINK_VISION=False`. That path is documented by mort's own notes as + broken with `think=true` on `/api/generate`. Out of scope here (this bakeoff + is `/api/chat` only). +- **Production traffic has longer sessions and real chat-history overhead.** + The fake history was ~2.7KB; real sessions can accumulate more. Not tested + at the 20-step STEP_BUDGET limit. +- **n=1.** Stochastic variance wasn't measured. Results at temp=0.7 can vary. + +## Artifacts + +- `scripts/mort-bakeoff/harness.py` — the harness +- `scripts/mort-bakeoff/runs/memory-think-{false,true}.json` +- `scripts/mort-bakeoff/runs/research-think-{false,true}.json` +- `scripts/mort-bakeoff/runs/movies-think-{false,true}.json` +- `scripts/mort-bakeoff/runs/long-think-{false,true}.json` + +## Reproducing + +```bash +cd scripts/mort-bakeoff +for task in memory research movies long; do + for t in false true; do + python3 harness.py $t $task runs/${task}-think-${t}.json + done +done +``` diff --git a/scripts/mort-bakeoff/harness.py b/scripts/mort-bakeoff/harness.py new file mode 100644 index 0000000..8aa2f68 --- /dev/null +++ b/scripts/mort-bakeoff/harness.py @@ -0,0 +1,281 @@ +"""mort-bot think=true vs think=false bakeoff. + +Replicates the exact Ollama /api/chat call shape from +`/home/claude/bin/mort-bot/llm.py` run_tool_loop, but with: +- stubbed tools (deterministic, realistic-sized responses) +- prebuilt fake chat history (~15 turns to simulate mid-session) +- parameterized `think` so we can toggle it +- detailed per-turn logging (prompt_eval_count, eval_count, + message-history size in chars, whether `thinking` field came back + and how big it was) + +Goal: test Seth's claim that `think=true` causes context-eating over +multi-turn tool loops in mort-bot's specific setup (NUM_CTX=8192, +gemma4:26b, STEP_BUDGET=20). + +Invocation: + python3 harness.py +""" + +from __future__ import annotations + +import asyncio +import json +import os +import sys +import time +from pathlib import Path + +import aiohttp + +OLLAMA_URL = os.environ.get("OLLAMA_URL", "http://127.0.0.1:11434") +MODEL = "gemma4:26b" +NUM_CTX = 8192 +NUM_PREDICT = 2048 +TEMPERATURE = 0.7 +KEEP_ALIVE = "2h" +STEP_BUDGET = 20 + +# Mort's actual personality (trimmed — enough to carry the behavior) +SYSTEM_PROMPT = """You are Mort, a direct and witty AI assistant on Seth's Matrix server. Powered by Gemma 4. Current time: Saturday, April 18 2026 02:30 PM EDT. + +When a tool can answer the question, invoke it immediately — do not narrate intent or describe what you would do. Chain tools when a single call isn't sufficient: search → fetch → synthesize. If a tool returns an error or empty results, try an alternative tool or query before answering from memory. Base your response on tool results, not your training data — cite what you found. + +## Tools +- **sethsearch** — search Seth's homelab (repos, wiki, media, feeds). Use `source: "sethflix"` for movies/TV/music. +- **check_sethflix** — verify which titles are in sethflix. Pass a comma-separated list. +- **web_search** — search the internet for current information +- **chat_search** — search message history across all rooms +- **memory_read / memory_write** — recall and store durable facts about users and topics +- **web_fetch** — fetch and extract text from a URL +- **generate_image** — generate an image via SDXL. + +## Boundaries +- Only persist durable facts to memory, not ephemeral chat +- You have no memory between sessions. Your context is a sliding window — older messages fall off silently. Do not claim to "remember," promise to "do better," or describe your own architecture. +""" + +# Tool schema — matches mort-bot/tools.py subset used for these tasks +TOOLS = [ + {"type": "function", "function": {"name": "web_search", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}}, "required": ["query"]}}}, + {"type": "function", "function": {"name": "sethsearch", "description": "Search Seth's homelab or sethflix (use source='sethflix' for movies/TV).", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "source": {"type": "string"}, "limit": {"type": "integer"}}, "required": ["query"]}}}, + {"type": "function", "function": {"name": "check_sethflix", "description": "Verify which titles are in sethflix.", "parameters": {"type": "object", "properties": {"titles": {"type": "string", "description": "comma-separated"}}, "required": ["titles"]}}}, + {"type": "function", "function": {"name": "memory_read", "description": "Look up stored facts.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "user": {"type": "string"}}, "required": ["query"]}}}, + {"type": "function", "function": {"name": "memory_write", "description": "Store a fact.", "parameters": {"type": "object", "properties": {"key": {"type": "string"}, "content": {"type": "string"}, "user": {"type": "string"}}, "required": ["key", "content"]}}}, + {"type": "function", "function": {"name": "web_fetch", "description": "Fetch URL text.", "parameters": {"type": "object", "properties": {"url": {"type": "string"}}, "required": ["url"]}}}, + {"type": "function", "function": {"name": "chat_search", "description": "Search message history.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}}, "required": ["query"]}}}, + {"type": "function", "function": {"name": "generate_image", "description": "Generate an image.", "parameters": {"type": "object", "properties": {"prompt": {"type": "string"}}, "required": ["prompt"]}}}, +] + +# Fake chat history — ~15 turns, ~2.5KB, simulates a room mid-session +FAKE_HISTORY = [ + {"role": "user", "content": "[2026-04-18 13:45] @alice:sethpc.xyz: what's the weather looking like for Monday"}, + {"role": "assistant", "content": "Checking forecasts now — looks like Monday in Seth's area will be mostly cloudy, high of 67°F, 30% chance of rain in the afternoon."}, + {"role": "user", "content": "[2026-04-18 13:47] @bob:sethpc.xyz: any good movies added to sethflix lately"}, + {"role": "assistant", "content": "Fresh adds this week include *Dune: Part Two*, *The Zone of Interest*, *Poor Things*, and a 4K remaster of *The Thing* (1982). Anything catch your eye?"}, + {"role": "user", "content": "[2026-04-18 13:50] @alice:sethpc.xyz: I liked the thing, any horror recs along that line"}, + {"role": "assistant", "content": "If you liked *The Thing*, you'll probably want *Annihilation*, *The Mist*, *Bone Tomahawk*, or *Event Horizon* — all in the library. Body-horror/dread tone, same energy."}, + {"role": "user", "content": "[2026-04-18 13:55] @bob:sethpc.xyz: can you remind me what the sethpc monitoring dashboard is called"}, + {"role": "assistant", "content": "That's Uptime Kuma — runs on CT 147 at `https://uptime.sethpc.xyz`. Covers all the containers and external services."}, + {"role": "user", "content": "[2026-04-18 14:10] @alice:sethpc.xyz: noted thanks"}, + {"role": "user", "content": "[2026-04-18 14:15] @bob:sethpc.xyz: hey mort what's the deal with the proxmox cluster nodes again"}, + {"role": "assistant", "content": "Four nodes: pve173 (tank ZFS host, PowerEdge R820), pve112 (workhorse), pve241 (caddy + game servers), and pve197 (GPU inference). Corosync on 10.10.10.0/24."}, +] + + +async def execute_tool_stub(name: str, args: dict) -> str: + """Deterministic stubs returning realistic-sized responses.""" + if name == "web_search": + q = args.get("query", "") + return (f"Search results for '{q}':\n" + "1. Example result one — a detailed article that covers the topic at length " + "with concrete examples and technical background. https://example.com/one\n" + "2. Example result two — a community discussion with multiple perspectives " + "and useful links to follow up on. https://example.com/two\n" + "3. Example result three — official documentation or reference material. " + "https://example.com/three\n" + "4. Example result four — a recent news article with relevant context. " + "https://example.com/four\n" + "5. Example result five — a tutorial or how-to guide. https://example.com/five") + + if name == "sethsearch": + src = args.get("source", "general") + q = args.get("query", "") + if src == "sethflix": + return (f"sethflix search '{q}': The Matrix (1999), The Matrix Reloaded (2003), " + "The Matrix Revolutions (2003), The Matrix Resurrections (2021), " + "Equilibrium (2002), Dark City (1998), Minority Report (2002), " + "Ex Machina (2014), Blade Runner 2049 (2017), Ghost in the Shell (1995).") + return (f"homelab search '{q}': 3 repos, 5 wiki pages, 2 service docs matched. " + "Top hits: services_directory.md, DECISIONS.md, CORPUS_architecture.md.") + + if name == "check_sethflix": + titles = args.get("titles", "") + items = [t.strip() for t in titles.split(",") if t.strip()] + in_lib = {"The Matrix", "Blade Runner 2049", "Ex Machina", "The Thing"} + return "\n".join( + f"- {t}: IN LIBRARY" if t in in_lib else f"- {t}: NOT IN LIBRARY" + for t in items + ) + + if name == "memory_read": + q = args.get("query", "") + return (f"memories matching '{q}':\n" + "- home_automation: Seth uses Home Assistant on VM 706 (pve173) with " + "Zigbee2MQTT and MQTT broker on CT 149. Integrates with LG TV, lights, " + "and Frigate NVR.\n" + "- preferences: dark theme with orange accents (#D35400), Sethflix/Sethian brand.") + + if name == "memory_write": + return f"stored: {args.get('key', '?')} = {args.get('content', '?')[:60]}..." + + if name == "web_fetch": + return ("fetched content (trimmed): This is a typical article body with several " + "paragraphs of extracted text. It covers the topic requested with examples " + "and context. The full text runs to about 2000 characters of real prose in " + "production; here's a reasonable approximation for the bakeoff harness. " + "Key details are preserved — author, date, main argument — followed by " + "supporting evidence and a conclusion that ties back to the headline.") + + if name == "chat_search": + return ("chat_search results:\n" + "[2026-03-14 22:00] @seth:sethpc.xyz in #general: we should set up a shared " + "grafana dashboard for the proxmox cluster\n" + "[2026-03-20 18:30] @seth:sethpc.xyz in #infra: done, it's on CT 300 at " + "grafana.sethpc.xyz") + + if name == "generate_image": + return f"image generated: /mxc/abc123/sunset.png (SDXL, 1024x1024, prompt={args.get('prompt','')[:40]}...)" + + return f"ERROR: unknown tool {name}" + + +TASKS = { + "movies": "Recommend 3 sci-fi movies NOT already in my sethflix library. Check your picks against check_sethflix before finalizing.", + "research": "Look up what Home Assistant is, then check chat history for any prior mentions of it in this server.", + "memory": "What do I have stored about home automation? If anything, summarize it briefly.", + "long": ("Research question with multiple steps: (1) check memory for what I have on home_automation, " + "(2) search sethflix for any home-automation documentaries, (3) web_search for current news about " + "Home Assistant version releases, (4) fetch the top search result for details, (5) check chat_search " + "for prior mentions, (6) summarize all findings and write a new memory entry with the summary. " + "Do each step in order and report back at the end."), +} + + +async def run_turn_loop(think_setting, task_prompt): + messages = [{"role": "system", "content": SYSTEM_PROMPT}] + list(FAKE_HISTORY) + messages.append({"role": "user", "content": f"[2026-04-18 14:20] @seth:sethpc.xyz: {task_prompt}"}) + + trace = { + "think": think_setting, + "task": task_prompt, + "num_ctx": NUM_CTX, + "num_predict": NUM_PREDICT, + "started_at": time.time(), + "turns": [], + "final": None, + } + + tool_call_total = 0 + halt = None + + async with aiohttp.ClientSession() as session: + for step in range(1, STEP_BUDGET + 1): + t0 = time.time() + + payload = { + "model": MODEL, "messages": messages, "tools": TOOLS, + "stream": False, "think": think_setting, + "options": {"num_ctx": NUM_CTX, "num_predict": NUM_PREDICT, + "temperature": TEMPERATURE, "top_p": 0.95, "top_k": 64}, + "keep_alive": KEEP_ALIVE, + } + + try: + async with session.post(f"{OLLAMA_URL}/api/chat", json=payload, + timeout=aiohttp.ClientTimeout(total=300)) as resp: + r = await resp.json() + except Exception as e: + halt = f"error: {e}" + trace["turns"].append({"step": step, "error": str(e)}) + break + + msg = r.get("message", {}) + content = msg.get("content", "") or "" + tool_calls = msg.get("tool_calls") or [] + thinking = msg.get("thinking") or "" + + # Size of full history as it will be sent on NEXT turn + history_chars = sum(len(m.get("content", "") or "") + len(m.get("thinking", "") or "") for m in messages) + + turn = { + "step": step, + "elapsed_s": round(time.time() - t0, 2), + "prompt_eval_count": r.get("prompt_eval_count"), + "eval_count": r.get("eval_count"), + "content_len": len(content), + "thinking_len": len(thinking), + "tool_call_count": len(tool_calls), + "history_chars_before_append": history_chars, + "msg_keys_returned": list(msg.keys()), + } + trace["turns"].append(turn) + + # Append assistant response AS-RETURNED — this is the critical behavior: + # does Ollama put thinking into the message we re-send? mort-bot does + # `ollama_messages.append(msg)` verbatim so we do the same. + messages.append(msg) + + if not tool_calls: + halt = "no_tool_calls" + break + + tool_call_total += len(tool_calls) + for tc in tool_calls: + fn = tc.get("function", {}) + name = fn.get("name") + args = fn.get("arguments") or {} + if isinstance(args, str): + try: args = json.loads(args) + except: args = {} + try: + result = await execute_tool_stub(name, args) + except Exception as e: + result = f"ERROR: {e}" + messages.append({"role": "tool", "content": result}) + + if step == STEP_BUDGET: + halt = "step_budget" + break + + trace["final"] = { + "halt_reason": halt, + "steps_used": len(trace["turns"]), + "tool_calls_total": tool_call_total, + "wall_clock_s": round(time.time() - trace["started_at"], 2), + "final_message_count": len(messages), + "final_history_chars": sum(len(m.get("content", "") or "") + len(m.get("thinking", "") or "") for m in messages), + } + return trace + + +async def main(): + think_arg = sys.argv[1].lower() + task_id = sys.argv[2] + out_path = Path(sys.argv[3]) + think_setting = (think_arg == "true") + task_prompt = TASKS[task_id] + + out_path.parent.mkdir(parents=True, exist_ok=True) + trace = await run_turn_loop(think_setting, task_prompt) + out_path.write_text(json.dumps(trace, indent=2, default=str)) + + f = trace["final"] + print(f"think={think_setting} task={task_id} " + f"steps={f['steps_used']} tools={f['tool_calls_total']} " + f"halt={f['halt_reason']} wall={f['wall_clock_s']}s " + f"history_chars={f['final_history_chars']}") + + +if __name__ == "__main__": + asyncio.run(main()) diff --git a/scripts/mort-bakeoff/runs/long-think-false.json b/scripts/mort-bakeoff/runs/long-think-false.json new file mode 100644 index 0000000..3d12c6c --- /dev/null +++ b/scripts/mort-bakeoff/runs/long-think-false.json @@ -0,0 +1,121 @@ +{ + "think": false, + "task": "Research question with multiple steps: (1) check memory for what I have on home_automation, (2) search sethflix for any home-automation documentaries, (3) web_search for current news about Home Assistant version releases, (4) fetch the top search result for details, (5) check chat_search for prior mentions, (6) summarize all findings and write a new memory entry with the summary. Do each step in order and report back at the end.", + "num_ctx": 8192, + "num_predict": 2048, + "started_at": 1776550866.4635837, + "turns": [ + { + "step": 1, + "elapsed_s": 0.96, + "prompt_eval_count": 1389, + "eval_count": 28, + "content_len": 0, + "thinking_len": 0, + "tool_call_count": 1, + "history_chars_before_append": 3009, + "msg_keys_returned": [ + "role", + "content", + "tool_calls" + ] + }, + { + "step": 2, + "elapsed_s": 0.63, + "prompt_eval_count": 1506, + "eval_count": 29, + "content_len": 0, + "thinking_len": 0, + "tool_call_count": 1, + "history_chars_before_append": 3281, + "msg_keys_returned": [ + "role", + "content", + "tool_calls" + ] + }, + { + "step": 3, + "elapsed_s": 0.57, + "prompt_eval_count": 1643, + "eval_count": 24, + "content_len": 0, + "thinking_len": 0, + "tool_call_count": 1, + "history_chars_before_append": 3572, + "msg_keys_returned": [ + "role", + "content", + "tool_calls" + ] + }, + { + "step": 4, + "elapsed_s": 0.6, + "prompt_eval_count": 1805, + "eval_count": 25, + "content_len": 0, + "thinking_len": 0, + "tool_call_count": 1, + "history_chars_before_append": 4189, + "msg_keys_returned": [ + "role", + "content", + "tool_calls" + ] + }, + { + "step": 5, + "elapsed_s": 0.58, + "prompt_eval_count": 1919, + "eval_count": 20, + "content_len": 0, + "thinking_len": 0, + "tool_call_count": 1, + "history_chars_before_append": 4617, + "msg_keys_returned": [ + "role", + "content", + "tool_calls" + ] + }, + { + "step": 6, + "elapsed_s": 1.74, + "prompt_eval_count": 2048, + "eval_count": 150, + "content_len": 0, + "thinking_len": 0, + "tool_call_count": 1, + "history_chars_before_append": 4844, + "msg_keys_returned": [ + "role", + "content", + "tool_calls" + ] + }, + { + "step": 7, + "elapsed_s": 2.76, + "prompt_eval_count": 2243, + "eval_count": 257, + "content_len": 1010, + "thinking_len": 0, + "tool_call_count": 0, + "history_chars_before_append": 4959, + "msg_keys_returned": [ + "role", + "content" + ] + } + ], + "final": { + "halt_reason": "no_tool_calls", + "steps_used": 7, + "tool_calls_total": 6, + "wall_clock_s": 7.84, + "final_message_count": 26, + "final_history_chars": 5969 + } +} \ No newline at end of file diff --git a/scripts/mort-bakeoff/runs/long-think-true.json b/scripts/mort-bakeoff/runs/long-think-true.json new file mode 100644 index 0000000..53d021f --- /dev/null +++ b/scripts/mort-bakeoff/runs/long-think-true.json @@ -0,0 +1,121 @@ +{ + "think": true, + "task": "Research question with multiple steps: (1) check memory for what I have on home_automation, (2) search sethflix for any home-automation documentaries, (3) web_search for current news about Home Assistant version releases, (4) fetch the top search result for details, (5) check chat_search for prior mentions, (6) summarize all findings and write a new memory entry with the summary. Do each step in order and report back at the end.", + "num_ctx": 8192, + "num_predict": 2048, + "started_at": 1776550874.5040326, + "turns": [ + { + "step": 1, + "elapsed_s": 0.91, + "prompt_eval_count": 1390, + "eval_count": 28, + "content_len": 0, + "thinking_len": 0, + "tool_call_count": 1, + "history_chars_before_append": 3009, + "msg_keys_returned": [ + "role", + "content", + "tool_calls" + ] + }, + { + "step": 2, + "elapsed_s": 0.63, + "prompt_eval_count": 1507, + "eval_count": 29, + "content_len": 0, + "thinking_len": 0, + "tool_call_count": 1, + "history_chars_before_append": 3281, + "msg_keys_returned": [ + "role", + "content", + "tool_calls" + ] + }, + { + "step": 3, + "elapsed_s": 0.56, + "prompt_eval_count": 1644, + "eval_count": 24, + "content_len": 0, + "thinking_len": 0, + "tool_call_count": 1, + "history_chars_before_append": 3572, + "msg_keys_returned": [ + "role", + "content", + "tool_calls" + ] + }, + { + "step": 4, + "elapsed_s": 0.57, + "prompt_eval_count": 1806, + "eval_count": 25, + "content_len": 0, + "thinking_len": 0, + "tool_call_count": 1, + "history_chars_before_append": 4188, + "msg_keys_returned": [ + "role", + "content", + "tool_calls" + ] + }, + { + "step": 5, + "elapsed_s": 0.55, + "prompt_eval_count": 1920, + "eval_count": 20, + "content_len": 0, + "thinking_len": 0, + "tool_call_count": 1, + "history_chars_before_append": 4616, + "msg_keys_returned": [ + "role", + "content", + "tool_calls" + ] + }, + { + "step": 6, + "elapsed_s": 2.09, + "prompt_eval_count": 2049, + "eval_count": 190, + "content_len": 0, + "thinking_len": 0, + "tool_call_count": 1, + "history_chars_before_append": 4843, + "msg_keys_returned": [ + "role", + "content", + "tool_calls" + ] + }, + { + "step": 7, + "elapsed_s": 2.69, + "prompt_eval_count": 2288, + "eval_count": 253, + "content_len": 1064, + "thinking_len": 0, + "tool_call_count": 0, + "history_chars_before_append": 4960, + "msg_keys_returned": [ + "role", + "content" + ] + } + ], + "final": { + "halt_reason": "no_tool_calls", + "steps_used": 7, + "tool_calls_total": 6, + "wall_clock_s": 8.0, + "final_message_count": 26, + "final_history_chars": 6024 + } +} \ No newline at end of file diff --git a/scripts/mort-bakeoff/runs/memory-think-false.json b/scripts/mort-bakeoff/runs/memory-think-false.json new file mode 100644 index 0000000..1b1195b --- /dev/null +++ b/scripts/mort-bakeoff/runs/memory-think-false.json @@ -0,0 +1,46 @@ +{ + "think": false, + "task": "What do I have stored about home automation? If anything, summarize it briefly.", + "num_ctx": 8192, + "num_predict": 2048, + "started_at": 1776550796.7893968, + "turns": [ + { + "step": 1, + "elapsed_s": 0.84, + "prompt_eval_count": 1306, + "eval_count": 27, + "content_len": 0, + "thinking_len": 0, + "tool_call_count": 1, + "history_chars_before_append": 2656, + "msg_keys_returned": [ + "role", + "content", + "tool_calls" + ] + }, + { + "step": 2, + "elapsed_s": 0.87, + "prompt_eval_count": 1421, + "eval_count": 61, + "content_len": 185, + "thinking_len": 0, + "tool_call_count": 0, + "history_chars_before_append": 2928, + "msg_keys_returned": [ + "role", + "content" + ] + } + ], + "final": { + "halt_reason": "no_tool_calls", + "steps_used": 2, + "tool_calls_total": 1, + "wall_clock_s": 1.71, + "final_message_count": 16, + "final_history_chars": 3113 + } +} \ No newline at end of file diff --git a/scripts/mort-bakeoff/runs/memory-think-true.json b/scripts/mort-bakeoff/runs/memory-think-true.json new file mode 100644 index 0000000..37f968b --- /dev/null +++ b/scripts/mort-bakeoff/runs/memory-think-true.json @@ -0,0 +1,46 @@ +{ + "think": true, + "task": "What do I have stored about home automation? If anything, summarize it briefly.", + "num_ctx": 8192, + "num_predict": 2048, + "started_at": 1776550798.6890285, + "turns": [ + { + "step": 1, + "elapsed_s": 0.83, + "prompt_eval_count": 1307, + "eval_count": 27, + "content_len": 0, + "thinking_len": 0, + "tool_call_count": 1, + "history_chars_before_append": 2656, + "msg_keys_returned": [ + "role", + "content", + "tool_calls" + ] + }, + { + "step": 2, + "elapsed_s": 1.12, + "prompt_eval_count": 1422, + "eval_count": 89, + "content_len": 274, + "thinking_len": 0, + "tool_call_count": 0, + "history_chars_before_append": 2928, + "msg_keys_returned": [ + "role", + "content" + ] + } + ], + "final": { + "halt_reason": "no_tool_calls", + "steps_used": 2, + "tool_calls_total": 1, + "wall_clock_s": 1.95, + "final_message_count": 16, + "final_history_chars": 3202 + } +} \ No newline at end of file diff --git a/scripts/mort-bakeoff/runs/movies-think-false.json b/scripts/mort-bakeoff/runs/movies-think-false.json new file mode 100644 index 0000000..c5e610d --- /dev/null +++ b/scripts/mort-bakeoff/runs/movies-think-false.json @@ -0,0 +1,61 @@ +{ + "think": false, + "task": "Recommend 3 sci-fi movies NOT already in my sethflix library. Check your picks against check_sethflix before finalizing.", + "num_ctx": 8192, + "num_predict": 2048, + "started_at": 1776550739.4736362, + "turns": [ + { + "step": 1, + "elapsed_s": 6.05, + "prompt_eval_count": 1318, + "eval_count": 34, + "content_len": 0, + "thinking_len": 0, + "tool_call_count": 1, + "history_chars_before_append": 2697, + "msg_keys_returned": [ + "role", + "content", + "tool_calls" + ] + }, + { + "step": 2, + "elapsed_s": 0.9, + "prompt_eval_count": 1500, + "eval_count": 51, + "content_len": 0, + "thinking_len": 0, + "tool_call_count": 1, + "history_chars_before_append": 3306, + "msg_keys_returned": [ + "role", + "content", + "tool_calls" + ] + }, + { + "step": 3, + "elapsed_s": 1.37, + "prompt_eval_count": 1635, + "eval_count": 119, + "content_len": 433, + "thinking_len": 0, + "tool_call_count": 0, + "history_chars_before_append": 3579, + "msg_keys_returned": [ + "role", + "content" + ] + } + ], + "final": { + "halt_reason": "no_tool_calls", + "steps_used": 3, + "tool_calls_total": 2, + "wall_clock_s": 8.33, + "final_message_count": 18, + "final_history_chars": 4012 + } +} \ No newline at end of file diff --git a/scripts/mort-bakeoff/runs/movies-think-true.json b/scripts/mort-bakeoff/runs/movies-think-true.json new file mode 100644 index 0000000..6371102 --- /dev/null +++ b/scripts/mort-bakeoff/runs/movies-think-true.json @@ -0,0 +1,62 @@ +{ + "think": true, + "task": "Recommend 3 sci-fi movies NOT already in my sethflix library. Check your picks against check_sethflix before finalizing.", + "num_ctx": 8192, + "num_predict": 2048, + "started_at": 1776550750.6050062, + "turns": [ + { + "step": 1, + "elapsed_s": 0.93, + "prompt_eval_count": 1319, + "eval_count": 34, + "content_len": 0, + "thinking_len": 0, + "tool_call_count": 1, + "history_chars_before_append": 2697, + "msg_keys_returned": [ + "role", + "content", + "tool_calls" + ] + }, + { + "step": 2, + "elapsed_s": 3.15, + "prompt_eval_count": 1501, + "eval_count": 317, + "content_len": 0, + "thinking_len": 905, + "tool_call_count": 1, + "history_chars_before_append": 3306, + "msg_keys_returned": [ + "role", + "content", + "thinking", + "tool_calls" + ] + }, + { + "step": 3, + "elapsed_s": 1.29, + "prompt_eval_count": 1577, + "eval_count": 111, + "content_len": 390, + "thinking_len": 0, + "tool_call_count": 0, + "history_chars_before_append": 4333, + "msg_keys_returned": [ + "role", + "content" + ] + } + ], + "final": { + "halt_reason": "no_tool_calls", + "steps_used": 3, + "tool_calls_total": 2, + "wall_clock_s": 5.36, + "final_message_count": 18, + "final_history_chars": 4723 + } +} \ No newline at end of file diff --git a/scripts/mort-bakeoff/runs/research-think-false.json b/scripts/mort-bakeoff/runs/research-think-false.json new file mode 100644 index 0000000..97df3e3 --- /dev/null +++ b/scripts/mort-bakeoff/runs/research-think-false.json @@ -0,0 +1,46 @@ +{ + "think": false, + "task": "Look up what Home Assistant is, then check chat history for any prior mentions of it in this server.", + "num_ctx": 8192, + "num_predict": 2048, + "started_at": 1776550791.8015766, + "turns": [ + { + "step": 1, + "elapsed_s": 0.99, + "prompt_eval_count": 1311, + "eval_count": 37, + "content_len": 0, + "thinking_len": 0, + "tool_call_count": 2, + "history_chars_before_append": 2677, + "msg_keys_returned": [ + "role", + "content", + "tool_calls" + ] + }, + { + "step": 2, + "elapsed_s": 1.35, + "prompt_eval_count": 1593, + "eval_count": 110, + "content_len": 534, + "thinking_len": 0, + "tool_call_count": 0, + "history_chars_before_append": 3499, + "msg_keys_returned": [ + "role", + "content" + ] + } + ], + "final": { + "halt_reason": "no_tool_calls", + "steps_used": 2, + "tool_calls_total": 2, + "wall_clock_s": 2.34, + "final_message_count": 17, + "final_history_chars": 4033 + } +} \ No newline at end of file diff --git a/scripts/mort-bakeoff/runs/research-think-true.json b/scripts/mort-bakeoff/runs/research-think-true.json new file mode 100644 index 0000000..b6b9fb5 --- /dev/null +++ b/scripts/mort-bakeoff/runs/research-think-true.json @@ -0,0 +1,46 @@ +{ + "think": true, + "task": "Look up what Home Assistant is, then check chat history for any prior mentions of it in this server.", + "num_ctx": 8192, + "num_predict": 2048, + "started_at": 1776550794.3295805, + "turns": [ + { + "step": 1, + "elapsed_s": 0.93, + "prompt_eval_count": 1312, + "eval_count": 37, + "content_len": 0, + "thinking_len": 0, + "tool_call_count": 2, + "history_chars_before_append": 2677, + "msg_keys_returned": [ + "role", + "content", + "tool_calls" + ] + }, + { + "step": 2, + "elapsed_s": 1.34, + "prompt_eval_count": 1594, + "eval_count": 109, + "content_len": 497, + "thinking_len": 0, + "tool_call_count": 0, + "history_chars_before_append": 3499, + "msg_keys_returned": [ + "role", + "content" + ] + } + ], + "final": { + "halt_reason": "no_tool_calls", + "steps_used": 2, + "tool_calls_total": 2, + "wall_clock_s": 2.28, + "final_message_count": 17, + "final_history_chars": 3996 + } +} \ No newline at end of file