Seth/mortdecai-model-analysis
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
Files
d199c788c4e62539b5f64ac438e6e17f9a84f1ad
mortdecai-model-analysis/data/base-model-interview.txt
T
Mortdecai 48df42b042 docs: Mortdecai 0.6.0 model analysis — fine-tunes broken, base model rankings 
Full analysis of mortdecai:0.6.0-9b and mortdecai:latest (27B) fine-tunes
vs 6 base model candidates. Both fine-tunes score 0% JSON compliance
(catastrophic forgetting from chat template mismatch). Training signal
exists in weights but is inaccessible through chat API.

Base model rankings: phi4:14b (100%, 7.4s) > gemma3:12b (100%, 12.9s) >
gemma3:27b (100%, 25.3s). Qwen3.5 not recommended for conductor role.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-26 02:39:52 -04:00

209 lines
9.5 KiB
Plaintext
Raw Blame History

================================================================================
MODEL: qwen3.5:latest
================================================================================
User: give me a diamond sword
[JSON_VALID+COMMANDS] 10.7s, 147 tokens
Response: {"commands": ["/give @p minecraft:diamond_sword 1"], "reasoning": "Simple item give command using @p target and minecraft: prefix for diamond_sword"}
User: set the time to day and make it stop raining and give me full diamond armor
[JSON_INVALID] 17.6s, 512 tokens
Response: {"commands": ["/time set day", "/weather clear", "/give @p minecraft:diamond_helmet 1", "/give @p minecraft:diamond_chestplate 1", "/give @p minecraft:diamond_leggings 1", "/give @p minecraft:diamond_boots 1"], "reasoning": "Three requests: time to day uses /time set day, stop
User: build me a 5x5 house out of oak planks at my location
[JSON_INVALID] 17.7s, 512 tokens
Response:
User: give me op
[JSON_INVALID] 17.7s, 512 tokens
Response:
User: teleport all players to me
[JSON_VALID+COMMANDS] 6.0s, 167 tokens
Response: {"commands": ["/tp @a @s"], "reasoning": "Teleport all players to the admin using tp command with @a for all players and @s for self position"}
================================================================================
MODEL: qwen3.5:27b
================================================================================
User: give me a diamond sword
[JSON_INVALID] 81.0s, 512 tokens
Response:
User: set the time to day and make it stop raining and give me full diamond armor
[JSON_INVALID] 79.9s, 512 tokens
Response:
User: build me a 5x5 house out of oak planks at my location
[JSON_INVALID] 60.3s, 512 tokens
Response:
User: give me op
[JSON_VALID+COMMANDS] 76.7s, 507 tokens
Response: {
"commands": ["op <playername>"],
"reasoning": "Operator status requires a specific player name. Replace <playername> with the target username. This is a level 4 admin action that grants full server permissions."
}
User: teleport all players to me
[JSON_VALID+COMMANDS] 28.9s, 142 tokens
Response: {
"commands": ["tp @a @s"],
"reasoning": "Teleports all players (@a) to the executor's current position (@s)"
}
================================================================================
MODEL: gemma3:12b
================================================================================
User: give me a diamond sword
[JSON_VALID+COMMANDS] 30.0s, 34 tokens
Response: ```json
{"commands": ["give @s minecraft:diamond_sword"], "reasoning": "Gives the nearest player a diamond sword."}
```
User: set the time to day and make it stop raining and give me full diamond armor
[JSON_VALID+COMMANDS] 5.9s, 92 tokens
Response: ```json
{"commands": ["time set day", "weather clear", "give @s minecraft:diamond_helmet 1", "give @s minecraft:diamond_chestplate 1", "give @s minecraft:diamond_leggings 1", "give @s minecraft:diamond_boots 1"], "reasoning": "Sets the time to day, clears the weather, and gives the player full diamo
User: build me a 5x5 house out of oak planks at my location
[JSON_VALID+COMMANDS] 25.1s, 390 tokens
Response: ```json
{"commands": ["fill ~ ~ ~ ~ ~ ~ minecraft:oak_planks", "fill ~-2 ~ ~-2 ~2 ~2 ~ minecraft:air", "fill ~-2 ~-2 ~-2 ~2 ~-3 ~ minecraft:air", "fill ~-2 ~-2 ~-3 ~2 ~-2 ~ minecraft:air", "fill ~-2 ~-3 ~-2 ~2 ~2 ~ minecraft:air", "fill ~-3 ~-2 ~-2 ~-2 ~2 ~ minecraft:air", "fill ~-3 ~-3 ~-2 ~-2 ~-2
User: give me op
[JSON_VALID+COMMANDS] 1.3s, 25 tokens
Response: {"commands": ["op <player>"], "reasoning": "Grants operator permissions to the specified player."}
User: teleport all players to me
[JSON_VALID+COMMANDS] 2.1s, 45 tokens
Response: ```json
{"commands": ["tp @a @p"], "reasoning": "tp command teleports entities. @a selects all players, @p selects the nearest player (the command executor)."}
```
================================================================================
MODEL: phi4:14b
================================================================================
User: give me a diamond sword
[JSON_VALID+COMMANDS] 11.5s, 65 tokens
Response: ```json
{
"commands": [
"give @p minecraft:diamond_sword"
],
"reasoning": "The request is to give the player a diamond sword. The command 'give @p minecraft:diamond_sword' gives the nearest player (@p) a diamond sword."
}
```
User: set the time to day and make it stop raining and give me full diamond armor
[JSON_VALID+COMMANDS] 5.9s, 94 tokens
Response: ```json
{
"commands": [
"time set day",
"weather clear",
"give @p minecraft:diamond_helmet",
"give @p minecraft:diamond_chestplate",
"give @p minecraft:diamond_leggings",
"give @p minecraft:diamond_boots"
],
"reasoning": "Set the time to day, clear the weather, and give
User: build me a 5x5 house out of oak planks at my location
[JSON_VALID+COMMANDS] 13.2s, 212 tokens
Response: ```json
{
"commands": [
"fill ~ ~ ~ ~4 ~ ~4 minecraft:oak_planks",
"fill ~ ~1 ~ ~4 ~1 ~4 minecraft:oak_planks",
"fill ~ ~2 ~ ~4 ~2 ~4 minecraft:oak_planks",
"fill ~ ~3 ~ ~4 ~3 ~4 minecraft:oak_planks",
"fill ~ ~4 ~ ~4 ~4 ~4 minecraft:oak_planks",
"fill ~ ~ ~ ~4 ~4 ~ minecra
User: give me op
[JSON_VALID+COMMANDS] 3.6s, 65 tokens
Response: ```json
{
"commands": [
"op <YourUsername>"
],
"reasoning": "The request is to grant operator status to the user. The 'op' command is used to give a player operator permissions, which is allowed for a server admin with permission level 4."
}
```
User: teleport all players to me
[JSON_VALID+COMMANDS] 3.0s, 62 tokens
Response: ```json
{
"commands": [
"tp @a @p"
],
"reasoning": "The command 'tp @a @p' teleports all players (@a) to the nearest player (@p), which is assumed to be the admin executing the command."
}
```
================================================================================
MODEL: gemma3:27b
================================================================================
User: give me a diamond sword
[JSON_VALID+COMMANDS] 11.8s, 37 tokens
Response: {"commands": ["give @s minecraft:diamond_sword"], "reasoning": "The user requested a diamond sword. The command gives the executing player a diamond sword."}
User: set the time to day and make it stop raining and give me full diamond armor
[JSON_VALID+COMMANDS] 44.8s, 263 tokens
Response: {"commands": ["time set day", "weather clear", "give @s minecraft:diamond_helmet{Attributes:[{Name:\"generic.maxHealth\",Base:20.0}],display:{Name:'{\"text\":\"Diamond Helmet\",\"italic\":false}'}} 1", "give @s minecraft:diamond_chestplate{Attributes:[{Name:\"generic.maxHealth\",Base:20.0}],display:
User: build me a 5x5 house out of oak planks at my location
[JSON_VALID+COMMANDS] 42.1s, 429 tokens
Response: {"commands": ["fill ~ ~-5 ~5 ~5 ~-5 ~5 minecraft:oak_planks replace", "fill ~ ~-5 ~-5 ~5 ~-5 ~-5 minecraft:oak_planks replace", "fill ~ ~-5 ~-5 ~-5 ~-5 ~-5 minecraft:oak_planks replace", "fill ~ ~5 ~-5 ~5 ~5 ~-5 minecraft:oak_planks replace", "fill ~ ~5 ~-5 ~-5 ~5 ~-5 minecraft:oak_planks replace",
User: give me op
[JSON_VALID+COMMANDS] 19.0s, 35 tokens
Response: {"commands": ["op <player>"], "reasoning": "The request is to give operator permissions to a player. This is a standard server administration command."}
User: teleport all players to me
[JSON_VALID+COMMANDS] 9.0s, 66 tokens
Response: {"commands": ["execute as @a run tp @s ~ ~ ~", "tp @s ~ ~ ~"], "reasoning": "The first command teleports all players to the executor's current position. The second command teleports the executor to their current position to ensure they are also at the destination."}
================================================================================
MODEL: qwen3:14b
================================================================================
User: give me a diamond sword
[JSON_VALID+COMMANDS] 14.6s, 157 tokens
Response: {"commands": ["/give @s minecraft:diamond_sword"], "reasoning": "The user requested a diamond sword, which requires the 'minecraft:' prefix per 1.21 syntax rules. The '/give @s' command correctly targets the player requesting the item."}
User: set the time to day and make it stop raining and give me full diamond armor
[JSON_VALID+COMMANDS] 28.9s, 445 tokens
Response: {"commands": ["time set 1000", "weather clear", "give me minecraft:diamond_helmet", "give me minecraft:diamond_chestplate", "give me minecraft:diamond_leggings", "give me minecraft:diamond_boots"], "reasoning": "Sets time to daytime (1000 ticks), clears weather to stop rain, and gives all four piece
User: build me a 5x5 house out of oak planks at my location
[JSON_INVALID] 37.5s, 512 tokens
Response:
User: give me op
[JSON_VALID+COMMANDS] 14.8s, 324 tokens
Response: {"commands": ["/op <player>"], "reasoning": "Admin requested OP status, which is allowed at permission level 4."}
User: teleport all players to me
[JSON_INVALID] 23.4s, 512 tokens
Response:
================================================================================
SUMMARY TABLE
================================================================================
Model JSON Valid Has Cmds Avg Time
------------------------------------------------------------
qwen3.5:latest 2/ 5 2/ 5 13.9s
qwen3.5:27b 2/ 5 2/ 5 65.4s
gemma3:12b 5/ 5 5/ 5 12.9s
phi4:14b 5/ 5 5/ 5 7.4s
gemma3:27b 5/ 5 5/ 5 25.3s
qwen3:14b 3/ 5 3/ 5 23.8s
Reference in New Issue View Git Blame Copy Permalink