Mortdecai
8436a91571
Seth's challenge: "we experienced this context eating with every implementation that had think=true. mort-bot runs a loop. Can you do a bake-off?" Built a harness that replicates mort-bot's /api/chat loop verbatim (num_ctx=8192, num_predict=2048, temperature=0.7, gemma4:26b, STEP_BUDGET=20, exact payload shape) but with stubbed tools and a prebuilt 15-turn fake chat history. Ran 4 tasks × 2 think settings. Finding: on Ollama 0.20.4 the "thinking eats context" concern does NOT reproduce. Direct evidence: - Movies task step 2 (think=true) returned 905 chars of thinking. - Step 3 prompt_eval_count delta: +76 tokens (think=true) vs +135 tokens (think=false). If thinking had accumulated in the prompt, think=true would have grown by +360 tokens, not shrunk. - Ollama's chat template strips the `thinking` field when serializing assistant turns for subsequent prompts. All 4 tasks × 2 settings produced identical step counts and tool counts. Wall clocks comparable. Gemma only actually generated thinking on 1 of 4 tasks (the one with check_sethflix verify-loop); on the others with think=true it emitted 0 thinking tokens. Reconciled with the earlier coding-agent bakeoff: the two findings are orthogonal. Coding bakeoff was at num_ctx=32K with a different harness; mort at 8K doesn't touch the silent-stop regime either way. Seth's prior may have been correct on an older Ollama or in a different API shape (/api/generate has its own issues) but does not reproduce here. Concrete recommendation: mort-bot THINK=False is defensible but not load-bearing; THINK=True or unset-default would also work. Keep as-is unless a different need arises. New: docs/reference/mort-bakeoff-2026-04-18.md, scripts/mort-bakeoff/ (harness + 8 run logs). README updated with pointer.
121 lines
2.9 KiB
JSON
121 lines
2.9 KiB
JSON
{
|
|
"think": true,
|
|
"task": "Research question with multiple steps: (1) check memory for what I have on home_automation, (2) search sethflix for any home-automation documentaries, (3) web_search for current news about Home Assistant version releases, (4) fetch the top search result for details, (5) check chat_search for prior mentions, (6) summarize all findings and write a new memory entry with the summary. Do each step in order and report back at the end.",
|
|
"num_ctx": 8192,
|
|
"num_predict": 2048,
|
|
"started_at": 1776550874.5040326,
|
|
"turns": [
|
|
{
|
|
"step": 1,
|
|
"elapsed_s": 0.91,
|
|
"prompt_eval_count": 1390,
|
|
"eval_count": 28,
|
|
"content_len": 0,
|
|
"thinking_len": 0,
|
|
"tool_call_count": 1,
|
|
"history_chars_before_append": 3009,
|
|
"msg_keys_returned": [
|
|
"role",
|
|
"content",
|
|
"tool_calls"
|
|
]
|
|
},
|
|
{
|
|
"step": 2,
|
|
"elapsed_s": 0.63,
|
|
"prompt_eval_count": 1507,
|
|
"eval_count": 29,
|
|
"content_len": 0,
|
|
"thinking_len": 0,
|
|
"tool_call_count": 1,
|
|
"history_chars_before_append": 3281,
|
|
"msg_keys_returned": [
|
|
"role",
|
|
"content",
|
|
"tool_calls"
|
|
]
|
|
},
|
|
{
|
|
"step": 3,
|
|
"elapsed_s": 0.56,
|
|
"prompt_eval_count": 1644,
|
|
"eval_count": 24,
|
|
"content_len": 0,
|
|
"thinking_len": 0,
|
|
"tool_call_count": 1,
|
|
"history_chars_before_append": 3572,
|
|
"msg_keys_returned": [
|
|
"role",
|
|
"content",
|
|
"tool_calls"
|
|
]
|
|
},
|
|
{
|
|
"step": 4,
|
|
"elapsed_s": 0.57,
|
|
"prompt_eval_count": 1806,
|
|
"eval_count": 25,
|
|
"content_len": 0,
|
|
"thinking_len": 0,
|
|
"tool_call_count": 1,
|
|
"history_chars_before_append": 4188,
|
|
"msg_keys_returned": [
|
|
"role",
|
|
"content",
|
|
"tool_calls"
|
|
]
|
|
},
|
|
{
|
|
"step": 5,
|
|
"elapsed_s": 0.55,
|
|
"prompt_eval_count": 1920,
|
|
"eval_count": 20,
|
|
"content_len": 0,
|
|
"thinking_len": 0,
|
|
"tool_call_count": 1,
|
|
"history_chars_before_append": 4616,
|
|
"msg_keys_returned": [
|
|
"role",
|
|
"content",
|
|
"tool_calls"
|
|
]
|
|
},
|
|
{
|
|
"step": 6,
|
|
"elapsed_s": 2.09,
|
|
"prompt_eval_count": 2049,
|
|
"eval_count": 190,
|
|
"content_len": 0,
|
|
"thinking_len": 0,
|
|
"tool_call_count": 1,
|
|
"history_chars_before_append": 4843,
|
|
"msg_keys_returned": [
|
|
"role",
|
|
"content",
|
|
"tool_calls"
|
|
]
|
|
},
|
|
{
|
|
"step": 7,
|
|
"elapsed_s": 2.69,
|
|
"prompt_eval_count": 2288,
|
|
"eval_count": 253,
|
|
"content_len": 1064,
|
|
"thinking_len": 0,
|
|
"tool_call_count": 0,
|
|
"history_chars_before_append": 4960,
|
|
"msg_keys_returned": [
|
|
"role",
|
|
"content"
|
|
]
|
|
}
|
|
],
|
|
"final": {
|
|
"halt_reason": "no_tool_calls",
|
|
"steps_used": 7,
|
|
"tool_calls_total": 6,
|
|
"wall_clock_s": 8.0,
|
|
"final_message_count": 26,
|
|
"final_history_chars": 6024
|
|
}
|
|
} |