Mortdecai
8436a91571
Seth's challenge: "we experienced this context eating with every implementation that had think=true. mort-bot runs a loop. Can you do a bake-off?" Built a harness that replicates mort-bot's /api/chat loop verbatim (num_ctx=8192, num_predict=2048, temperature=0.7, gemma4:26b, STEP_BUDGET=20, exact payload shape) but with stubbed tools and a prebuilt 15-turn fake chat history. Ran 4 tasks × 2 think settings. Finding: on Ollama 0.20.4 the "thinking eats context" concern does NOT reproduce. Direct evidence: - Movies task step 2 (think=true) returned 905 chars of thinking. - Step 3 prompt_eval_count delta: +76 tokens (think=true) vs +135 tokens (think=false). If thinking had accumulated in the prompt, think=true would have grown by +360 tokens, not shrunk. - Ollama's chat template strips the `thinking` field when serializing assistant turns for subsequent prompts. All 4 tasks × 2 settings produced identical step counts and tool counts. Wall clocks comparable. Gemma only actually generated thinking on 1 of 4 tasks (the one with check_sethflix verify-loop); on the others with think=true it emitted 0 thinking tokens. Reconciled with the earlier coding-agent bakeoff: the two findings are orthogonal. Coding bakeoff was at num_ctx=32K with a different harness; mort at 8K doesn't touch the silent-stop regime either way. Seth's prior may have been correct on an older Ollama or in a different API shape (/api/generate has its own issues) but does not reproduce here. Concrete recommendation: mort-bot THINK=False is defensible but not load-bearing; THINK=True or unset-default would also work. Keep as-is unless a different need arises. New: docs/reference/mort-bakeoff-2026-04-18.md, scripts/mort-bakeoff/ (harness + 8 run logs). README updated with pointer.
46 lines
1.0 KiB
JSON
46 lines
1.0 KiB
JSON
{
|
|
"think": false,
|
|
"task": "Look up what Home Assistant is, then check chat history for any prior mentions of it in this server.",
|
|
"num_ctx": 8192,
|
|
"num_predict": 2048,
|
|
"started_at": 1776550791.8015766,
|
|
"turns": [
|
|
{
|
|
"step": 1,
|
|
"elapsed_s": 0.99,
|
|
"prompt_eval_count": 1311,
|
|
"eval_count": 37,
|
|
"content_len": 0,
|
|
"thinking_len": 0,
|
|
"tool_call_count": 2,
|
|
"history_chars_before_append": 2677,
|
|
"msg_keys_returned": [
|
|
"role",
|
|
"content",
|
|
"tool_calls"
|
|
]
|
|
},
|
|
{
|
|
"step": 2,
|
|
"elapsed_s": 1.35,
|
|
"prompt_eval_count": 1593,
|
|
"eval_count": 110,
|
|
"content_len": 534,
|
|
"thinking_len": 0,
|
|
"tool_call_count": 0,
|
|
"history_chars_before_append": 3499,
|
|
"msg_keys_returned": [
|
|
"role",
|
|
"content"
|
|
]
|
|
}
|
|
],
|
|
"final": {
|
|
"halt_reason": "no_tool_calls",
|
|
"steps_used": 2,
|
|
"tool_calls_total": 2,
|
|
"wall_clock_s": 2.34,
|
|
"final_message_count": 17,
|
|
"final_history_chars": 4033
|
|
}
|
|
} |