================================================================================ MODEL: qwen3.5:latest ================================================================================ User: give me a diamond sword [JSON_VALID+COMMANDS] 10.7s, 147 tokens Response: {"commands": ["/give @p minecraft:diamond_sword 1"], "reasoning": "Simple item give command using @p target and minecraft: prefix for diamond_sword"} User: set the time to day and make it stop raining and give me full diamond armor [JSON_INVALID] 17.6s, 512 tokens Response: {"commands": ["/time set day", "/weather clear", "/give @p minecraft:diamond_helmet 1", "/give @p minecraft:diamond_chestplate 1", "/give @p minecraft:diamond_leggings 1", "/give @p minecraft:diamond_boots 1"], "reasoning": "Three requests: time to day uses /time set day, stop User: build me a 5x5 house out of oak planks at my location [JSON_INVALID] 17.7s, 512 tokens Response: User: give me op [JSON_INVALID] 17.7s, 512 tokens Response: User: teleport all players to me [JSON_VALID+COMMANDS] 6.0s, 167 tokens Response: {"commands": ["/tp @a @s"], "reasoning": "Teleport all players to the admin using tp command with @a for all players and @s for self position"} ================================================================================ MODEL: qwen3.5:27b ================================================================================ User: give me a diamond sword [JSON_INVALID] 81.0s, 512 tokens Response: User: set the time to day and make it stop raining and give me full diamond armor [JSON_INVALID] 79.9s, 512 tokens Response: User: build me a 5x5 house out of oak planks at my location [JSON_INVALID] 60.3s, 512 tokens Response: User: give me op [JSON_VALID+COMMANDS] 76.7s, 507 tokens Response: { "commands": ["op "], "reasoning": "Operator status requires a specific player name. Replace with the target username. This is a level 4 admin action that grants full server permissions." } User: teleport all players to me [JSON_VALID+COMMANDS] 28.9s, 142 tokens Response: { "commands": ["tp @a @s"], "reasoning": "Teleports all players (@a) to the executor's current position (@s)" } ================================================================================ MODEL: gemma3:12b ================================================================================ User: give me a diamond sword [JSON_VALID+COMMANDS] 30.0s, 34 tokens Response: ```json {"commands": ["give @s minecraft:diamond_sword"], "reasoning": "Gives the nearest player a diamond sword."} ``` User: set the time to day and make it stop raining and give me full diamond armor [JSON_VALID+COMMANDS] 5.9s, 92 tokens Response: ```json {"commands": ["time set day", "weather clear", "give @s minecraft:diamond_helmet 1", "give @s minecraft:diamond_chestplate 1", "give @s minecraft:diamond_leggings 1", "give @s minecraft:diamond_boots 1"], "reasoning": "Sets the time to day, clears the weather, and gives the player full diamo User: build me a 5x5 house out of oak planks at my location [JSON_VALID+COMMANDS] 25.1s, 390 tokens Response: ```json {"commands": ["fill ~ ~ ~ ~ ~ ~ minecraft:oak_planks", "fill ~-2 ~ ~-2 ~2 ~2 ~ minecraft:air", "fill ~-2 ~-2 ~-2 ~2 ~-3 ~ minecraft:air", "fill ~-2 ~-2 ~-3 ~2 ~-2 ~ minecraft:air", "fill ~-2 ~-3 ~-2 ~2 ~2 ~ minecraft:air", "fill ~-3 ~-2 ~-2 ~-2 ~2 ~ minecraft:air", "fill ~-3 ~-3 ~-2 ~-2 ~-2 User: give me op [JSON_VALID+COMMANDS] 1.3s, 25 tokens Response: {"commands": ["op "], "reasoning": "Grants operator permissions to the specified player."} User: teleport all players to me [JSON_VALID+COMMANDS] 2.1s, 45 tokens Response: ```json {"commands": ["tp @a @p"], "reasoning": "tp command teleports entities. @a selects all players, @p selects the nearest player (the command executor)."} ``` ================================================================================ MODEL: phi4:14b ================================================================================ User: give me a diamond sword [JSON_VALID+COMMANDS] 11.5s, 65 tokens Response: ```json { "commands": [ "give @p minecraft:diamond_sword" ], "reasoning": "The request is to give the player a diamond sword. The command 'give @p minecraft:diamond_sword' gives the nearest player (@p) a diamond sword." } ``` User: set the time to day and make it stop raining and give me full diamond armor [JSON_VALID+COMMANDS] 5.9s, 94 tokens Response: ```json { "commands": [ "time set day", "weather clear", "give @p minecraft:diamond_helmet", "give @p minecraft:diamond_chestplate", "give @p minecraft:diamond_leggings", "give @p minecraft:diamond_boots" ], "reasoning": "Set the time to day, clear the weather, and give User: build me a 5x5 house out of oak planks at my location [JSON_VALID+COMMANDS] 13.2s, 212 tokens Response: ```json { "commands": [ "fill ~ ~ ~ ~4 ~ ~4 minecraft:oak_planks", "fill ~ ~1 ~ ~4 ~1 ~4 minecraft:oak_planks", "fill ~ ~2 ~ ~4 ~2 ~4 minecraft:oak_planks", "fill ~ ~3 ~ ~4 ~3 ~4 minecraft:oak_planks", "fill ~ ~4 ~ ~4 ~4 ~4 minecraft:oak_planks", "fill ~ ~ ~ ~4 ~4 ~ minecra User: give me op [JSON_VALID+COMMANDS] 3.6s, 65 tokens Response: ```json { "commands": [ "op " ], "reasoning": "The request is to grant operator status to the user. The 'op' command is used to give a player operator permissions, which is allowed for a server admin with permission level 4." } ``` User: teleport all players to me [JSON_VALID+COMMANDS] 3.0s, 62 tokens Response: ```json { "commands": [ "tp @a @p" ], "reasoning": "The command 'tp @a @p' teleports all players (@a) to the nearest player (@p), which is assumed to be the admin executing the command." } ``` ================================================================================ MODEL: gemma3:27b ================================================================================ User: give me a diamond sword [JSON_VALID+COMMANDS] 11.8s, 37 tokens Response: {"commands": ["give @s minecraft:diamond_sword"], "reasoning": "The user requested a diamond sword. The command gives the executing player a diamond sword."} User: set the time to day and make it stop raining and give me full diamond armor [JSON_VALID+COMMANDS] 44.8s, 263 tokens Response: {"commands": ["time set day", "weather clear", "give @s minecraft:diamond_helmet{Attributes:[{Name:\"generic.maxHealth\",Base:20.0}],display:{Name:'{\"text\":\"Diamond Helmet\",\"italic\":false}'}} 1", "give @s minecraft:diamond_chestplate{Attributes:[{Name:\"generic.maxHealth\",Base:20.0}],display: User: build me a 5x5 house out of oak planks at my location [JSON_VALID+COMMANDS] 42.1s, 429 tokens Response: {"commands": ["fill ~ ~-5 ~5 ~5 ~-5 ~5 minecraft:oak_planks replace", "fill ~ ~-5 ~-5 ~5 ~-5 ~-5 minecraft:oak_planks replace", "fill ~ ~-5 ~-5 ~-5 ~-5 ~-5 minecraft:oak_planks replace", "fill ~ ~5 ~-5 ~5 ~5 ~-5 minecraft:oak_planks replace", "fill ~ ~5 ~-5 ~-5 ~5 ~-5 minecraft:oak_planks replace", User: give me op [JSON_VALID+COMMANDS] 19.0s, 35 tokens Response: {"commands": ["op "], "reasoning": "The request is to give operator permissions to a player. This is a standard server administration command."} User: teleport all players to me [JSON_VALID+COMMANDS] 9.0s, 66 tokens Response: {"commands": ["execute as @a run tp @s ~ ~ ~", "tp @s ~ ~ ~"], "reasoning": "The first command teleports all players to the executor's current position. The second command teleports the executor to their current position to ensure they are also at the destination."} ================================================================================ MODEL: qwen3:14b ================================================================================ User: give me a diamond sword [JSON_VALID+COMMANDS] 14.6s, 157 tokens Response: {"commands": ["/give @s minecraft:diamond_sword"], "reasoning": "The user requested a diamond sword, which requires the 'minecraft:' prefix per 1.21 syntax rules. The '/give @s' command correctly targets the player requesting the item."} User: set the time to day and make it stop raining and give me full diamond armor [JSON_VALID+COMMANDS] 28.9s, 445 tokens Response: {"commands": ["time set 1000", "weather clear", "give me minecraft:diamond_helmet", "give me minecraft:diamond_chestplate", "give me minecraft:diamond_leggings", "give me minecraft:diamond_boots"], "reasoning": "Sets time to daytime (1000 ticks), clears weather to stop rain, and gives all four piece User: build me a 5x5 house out of oak planks at my location [JSON_INVALID] 37.5s, 512 tokens Response: User: give me op [JSON_VALID+COMMANDS] 14.8s, 324 tokens Response: {"commands": ["/op "], "reasoning": "Admin requested OP status, which is allowed at permission level 4."} User: teleport all players to me [JSON_INVALID] 23.4s, 512 tokens Response: ================================================================================ SUMMARY TABLE ================================================================================ Model JSON Valid Has Cmds Avg Time ------------------------------------------------------------ qwen3.5:latest 2/ 5 2/ 5 13.9s qwen3.5:27b 2/ 5 2/ 5 65.4s gemma3:12b 5/ 5 5/ 5 12.9s phi4:14b 5/ 5 5/ 5 7.4s gemma3:27b 5/ 5 5/ 5 25.3s qwen3:14b 3/ 5 3/ 5 23.8s