Seth/gemma4-research
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

feat: mort-bot think=true vs think=false bakeoff

Browse Source 
Seth's challenge: "we experienced this context eating with every
implementation that had think=true. mort-bot runs a loop. Can you do
a bake-off?"

Built a harness that replicates mort-bot's /api/chat loop verbatim
(num_ctx=8192, num_predict=2048, temperature=0.7, gemma4:26b,
STEP_BUDGET=20, exact payload shape) but with stubbed tools and a
prebuilt 15-turn fake chat history. Ran 4 tasks × 2 think settings.

Finding: on Ollama 0.20.4 the "thinking eats context" concern does
NOT reproduce. Direct evidence:
- Movies task step 2 (think=true) returned 905 chars of thinking.
- Step 3 prompt_eval_count delta: +76 tokens (think=true) vs +135
  tokens (think=false). If thinking had accumulated in the prompt,
  think=true would have grown by +360 tokens, not shrunk.
- Ollama's chat template strips the `thinking` field when serializing
  assistant turns for subsequent prompts.

All 4 tasks × 2 settings produced identical step counts and tool
counts. Wall clocks comparable. Gemma only actually generated
thinking on 1 of 4 tasks (the one with check_sethflix verify-loop);
on the others with think=true it emitted 0 thinking tokens.

Reconciled with the earlier coding-agent bakeoff: the two findings
are orthogonal. Coding bakeoff was at num_ctx=32K with a different
harness; mort at 8K doesn't touch the silent-stop regime either way.
Seth's prior may have been correct on an older Ollama or in a
different API shape (/api/generate has its own issues) but does not
reproduce here.

Concrete recommendation: mort-bot THINK=False is defensible but not
load-bearing; THINK=True or unset-default would also work. Keep as-is
unless a different need arises.

New: docs/reference/mort-bakeoff-2026-04-18.md, scripts/mort-bakeoff/
(harness + 8 run logs). README updated with pointer.
This commit is contained in:
Mortdecai
2026-04-18 18:23:43 -04:00
parent c61394923c
commit 8436a91571
12 changed files with 988 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files
scripts/mort-bakeoff/runs/long-think-false.json
+121
View File
@@ -0,0 +1,121 @@
{
"think": false,
"task": "Research question with multiple steps: (1) check memory for what I have on home_automation, (2) search sethflix for any home-automation documentaries, (3) web_search for current news about Home Assistant version releases, (4) fetch the top search result for details, (5) check chat_search for prior mentions, (6) summarize all findings and write a new memory entry with the summary. Do each step in order and report back at the end.",
"num_ctx": 8192,
"num_predict": 2048,
"started_at": 1776550866.4635837,
"turns": [
{
"step": 1,
"elapsed_s": 0.96,
"prompt_eval_count": 1389,
"eval_count": 28,
"content_len": 0,
"thinking_len": 0,
"tool_call_count": 1,
"history_chars_before_append": 3009,
"msg_keys_returned": [
"role",
"content",
"tool_calls"
]
},
{
"step": 2,
"elapsed_s": 0.63,
"prompt_eval_count": 1506,
"eval_count": 29,
"content_len": 0,
"thinking_len": 0,
"tool_call_count": 1,
"history_chars_before_append": 3281,
"msg_keys_returned": [
"role",
"content",
"tool_calls"
]
},
{
"step": 3,
"elapsed_s": 0.57,
"prompt_eval_count": 1643,
"eval_count": 24,
"content_len": 0,
"thinking_len": 0,
"tool_call_count": 1,
"history_chars_before_append": 3572,
"msg_keys_returned": [
"role",
"content",
"tool_calls"
]
},
{
"step": 4,
"elapsed_s": 0.6,
"prompt_eval_count": 1805,
"eval_count": 25,
"content_len": 0,
"thinking_len": 0,
"tool_call_count": 1,
"history_chars_before_append": 4189,
"msg_keys_returned": [
"role",
"content",
"tool_calls"
]
},
{
"step": 5,
"elapsed_s": 0.58,
"prompt_eval_count": 1919,
"eval_count": 20,
"content_len": 0,
"thinking_len": 0,
"tool_call_count": 1,
"history_chars_before_append": 4617,
"msg_keys_returned": [
"role",
"content",
"tool_calls"
]
},
{
"step": 6,
"elapsed_s": 1.74,
"prompt_eval_count": 2048,
"eval_count": 150,
"content_len": 0,
"thinking_len": 0,
"tool_call_count": 1,
"history_chars_before_append": 4844,
"msg_keys_returned": [
"role",
"content",
"tool_calls"
]
},
{
"step": 7,
"elapsed_s": 2.76,
"prompt_eval_count": 2243,
"eval_count": 257,
"content_len": 1010,
"thinking_len": 0,
"tool_call_count": 0,
"history_chars_before_append": 4959,
"msg_keys_returned": [
"role",
"content"
]
}
],
"final": {
"halt_reason": "no_tool_calls",
"steps_used": 7,
"tool_calls_total": 6,
"wall_clock_s": 7.84,
"final_message_count": 26,
"final_history_chars": 5969
}
}
scripts/mort-bakeoff/runs/long-think-true.json
+121
View File
@@ -0,0 +1,121 @@
{
"think": true,
"task": "Research question with multiple steps: (1) check memory for what I have on home_automation, (2) search sethflix for any home-automation documentaries, (3) web_search for current news about Home Assistant version releases, (4) fetch the top search result for details, (5) check chat_search for prior mentions, (6) summarize all findings and write a new memory entry with the summary. Do each step in order and report back at the end.",
"num_ctx": 8192,
"num_predict": 2048,
"started_at": 1776550874.5040326,
"turns": [
{
"step": 1,
"elapsed_s": 0.91,
"prompt_eval_count": 1390,
"eval_count": 28,
"content_len": 0,
"thinking_len": 0,
"tool_call_count": 1,
"history_chars_before_append": 3009,
"msg_keys_returned": [
"role",
"content",
"tool_calls"
]
},
{
"step": 2,
"elapsed_s": 0.63,
"prompt_eval_count": 1507,
"eval_count": 29,
"content_len": 0,
"thinking_len": 0,
"tool_call_count": 1,
"history_chars_before_append": 3281,
"msg_keys_returned": [
"role",
"content",
"tool_calls"
]
},
{
"step": 3,
"elapsed_s": 0.56,
"prompt_eval_count": 1644,
"eval_count": 24,
"content_len": 0,
"thinking_len": 0,
"tool_call_count": 1,
"history_chars_before_append": 3572,
"msg_keys_returned": [
"role",
"content",
"tool_calls"
]
},
{
"step": 4,
"elapsed_s": 0.57,
"prompt_eval_count": 1806,
"eval_count": 25,
"content_len": 0,
"thinking_len": 0,
"tool_call_count": 1,
"history_chars_before_append": 4188,
"msg_keys_returned": [
"role",
"content",
"tool_calls"
]
},
{
"step": 5,
"elapsed_s": 0.55,
"prompt_eval_count": 1920,
"eval_count": 20,
"content_len": 0,
"thinking_len": 0,
"tool_call_count": 1,
"history_chars_before_append": 4616,
"msg_keys_returned": [
"role",
"content",
"tool_calls"
]
},
{
"step": 6,
"elapsed_s": 2.09,
"prompt_eval_count": 2049,
"eval_count": 190,
"content_len": 0,
"thinking_len": 0,
"tool_call_count": 1,
"history_chars_before_append": 4843,
"msg_keys_returned": [
"role",
"content",
"tool_calls"
]
},
{
"step": 7,
"elapsed_s": 2.69,
"prompt_eval_count": 2288,
"eval_count": 253,
"content_len": 1064,
"thinking_len": 0,
"tool_call_count": 0,
"history_chars_before_append": 4960,
"msg_keys_returned": [
"role",
"content"
]
}
],
"final": {
"halt_reason": "no_tool_calls",
"steps_used": 7,
"tool_calls_total": 6,
"wall_clock_s": 8.0,
"final_message_count": 26,
"final_history_chars": 6024
}
}
scripts/mort-bakeoff/runs/memory-think-false.json
+46
View File
@@ -0,0 +1,46 @@
{
"think": false,
"task": "What do I have stored about home automation? If anything, summarize it briefly.",
"num_ctx": 8192,
"num_predict": 2048,
"started_at": 1776550796.7893968,
"turns": [
{
"step": 1,
"elapsed_s": 0.84,
"prompt_eval_count": 1306,
"eval_count": 27,
"content_len": 0,
"thinking_len": 0,
"tool_call_count": 1,
"history_chars_before_append": 2656,
"msg_keys_returned": [
"role",
"content",
"tool_calls"
]
},
{
"step": 2,
"elapsed_s": 0.87,
"prompt_eval_count": 1421,
"eval_count": 61,
"content_len": 185,
"thinking_len": 0,
"tool_call_count": 0,
"history_chars_before_append": 2928,
"msg_keys_returned": [
"role",
"content"
]
}
],
"final": {
"halt_reason": "no_tool_calls",
"steps_used": 2,
"tool_calls_total": 1,
"wall_clock_s": 1.71,
"final_message_count": 16,
"final_history_chars": 3113
}
}
scripts/mort-bakeoff/runs/memory-think-true.json
+46
View File
@@ -0,0 +1,46 @@
{
"think": true,
"task": "What do I have stored about home automation? If anything, summarize it briefly.",
"num_ctx": 8192,
"num_predict": 2048,
"started_at": 1776550798.6890285,
"turns": [
{
"step": 1,
"elapsed_s": 0.83,
"prompt_eval_count": 1307,
"eval_count": 27,
"content_len": 0,
"thinking_len": 0,
"tool_call_count": 1,
"history_chars_before_append": 2656,
"msg_keys_returned": [
"role",
"content",
"tool_calls"
]
},
{
"step": 2,
"elapsed_s": 1.12,
"prompt_eval_count": 1422,
"eval_count": 89,
"content_len": 274,
"thinking_len": 0,
"tool_call_count": 0,
"history_chars_before_append": 2928,
"msg_keys_returned": [
"role",
"content"
]
}
],
"final": {
"halt_reason": "no_tool_calls",
"steps_used": 2,
"tool_calls_total": 1,
"wall_clock_s": 1.95,
"final_message_count": 16,
"final_history_chars": 3202
}
}
scripts/mort-bakeoff/runs/movies-think-false.json
+61
View File
@@ -0,0 +1,61 @@
{
"think": false,
"task": "Recommend 3 sci-fi movies NOT already in my sethflix library. Check your picks against check_sethflix before finalizing.",
"num_ctx": 8192,
"num_predict": 2048,
"started_at": 1776550739.4736362,
"turns": [
{
"step": 1,
"elapsed_s": 6.05,
"prompt_eval_count": 1318,
"eval_count": 34,
"content_len": 0,
"thinking_len": 0,
"tool_call_count": 1,
"history_chars_before_append": 2697,
"msg_keys_returned": [
"role",
"content",
"tool_calls"
]
},
{
"step": 2,
"elapsed_s": 0.9,
"prompt_eval_count": 1500,
"eval_count": 51,
"content_len": 0,
"thinking_len": 0,
"tool_call_count": 1,
"history_chars_before_append": 3306,
"msg_keys_returned": [
"role",
"content",
"tool_calls"
]
},
{
"step": 3,
"elapsed_s": 1.37,
"prompt_eval_count": 1635,
"eval_count": 119,
"content_len": 433,
"thinking_len": 0,
"tool_call_count": 0,
"history_chars_before_append": 3579,
"msg_keys_returned": [
"role",
"content"
]
}
],
"final": {
"halt_reason": "no_tool_calls",
"steps_used": 3,
"tool_calls_total": 2,
"wall_clock_s": 8.33,
"final_message_count": 18,
"final_history_chars": 4012
}
}
scripts/mort-bakeoff/runs/movies-think-true.json
+62
View File
@@ -0,0 +1,62 @@
{
"think": true,
"task": "Recommend 3 sci-fi movies NOT already in my sethflix library. Check your picks against check_sethflix before finalizing.",
"num_ctx": 8192,
"num_predict": 2048,
"started_at": 1776550750.6050062,
"turns": [
{
"step": 1,
"elapsed_s": 0.93,
"prompt_eval_count": 1319,
"eval_count": 34,
"content_len": 0,
"thinking_len": 0,
"tool_call_count": 1,
"history_chars_before_append": 2697,
"msg_keys_returned": [
"role",
"content",
"tool_calls"
]
},
{
"step": 2,
"elapsed_s": 3.15,
"prompt_eval_count": 1501,
"eval_count": 317,
"content_len": 0,
"thinking_len": 905,
"tool_call_count": 1,
"history_chars_before_append": 3306,
"msg_keys_returned": [
"role",
"content",
"thinking",
"tool_calls"
]
},
{
"step": 3,
"elapsed_s": 1.29,
"prompt_eval_count": 1577,
"eval_count": 111,
"content_len": 390,
"thinking_len": 0,
"tool_call_count": 0,
"history_chars_before_append": 4333,
"msg_keys_returned": [
"role",
"content"
]
}
],
"final": {
"halt_reason": "no_tool_calls",
"steps_used": 3,
"tool_calls_total": 2,
"wall_clock_s": 5.36,
"final_message_count": 18,
"final_history_chars": 4723
}
}
scripts/mort-bakeoff/runs/research-think-false.json
+46
View File
@@ -0,0 +1,46 @@
{
"think": false,
"task": "Look up what Home Assistant is, then check chat history for any prior mentions of it in this server.",
"num_ctx": 8192,
"num_predict": 2048,
"started_at": 1776550791.8015766,
"turns": [
{
"step": 1,
"elapsed_s": 0.99,
"prompt_eval_count": 1311,
"eval_count": 37,
"content_len": 0,
"thinking_len": 0,
"tool_call_count": 2,
"history_chars_before_append": 2677,
"msg_keys_returned": [
"role",
"content",
"tool_calls"
]
},
{
"step": 2,
"elapsed_s": 1.35,
"prompt_eval_count": 1593,
"eval_count": 110,
"content_len": 534,
"thinking_len": 0,
"tool_call_count": 0,
"history_chars_before_append": 3499,
"msg_keys_returned": [
"role",
"content"
]
}
],
"final": {
"halt_reason": "no_tool_calls",
"steps_used": 2,
"tool_calls_total": 2,
"wall_clock_s": 2.34,
"final_message_count": 17,
"final_history_chars": 4033
}
}
scripts/mort-bakeoff/runs/research-think-true.json
+46
View File
@@ -0,0 +1,46 @@
{
"think": true,
"task": "Look up what Home Assistant is, then check chat history for any prior mentions of it in this server.",
"num_ctx": 8192,
"num_predict": 2048,
"started_at": 1776550794.3295805,
"turns": [
{
"step": 1,
"elapsed_s": 0.93,
"prompt_eval_count": 1312,
"eval_count": 37,
"content_len": 0,
"thinking_len": 0,
"tool_call_count": 2,
"history_chars_before_append": 2677,
"msg_keys_returned": [
"role",
"content",
"tool_calls"
]
},
{
"step": 2,
"elapsed_s": 1.34,
"prompt_eval_count": 1594,
"eval_count": 109,
"content_len": 497,
"thinking_len": 0,
"tool_call_count": 0,
"history_chars_before_append": 3499,
"msg_keys_returned": [
"role",
"content"
]
}
],
"final": {
"halt_reason": "no_tool_calls",
"steps_used": 2,
"tool_calls_total": 2,
"wall_clock_s": 2.28,
"final_message_count": 17,
"final_history_chars": 3996
}
}