GPT-6 Luna

The scorecard

Each axis runs from 0 to 100. The colored mark is this model. The faint marks are the other models in this edition. Hover one to see which, and click it to open that model.

• Tool use

  The agent has to call the right tool, with the right arguments, in the right order. Telling the customer it's done before the tool has finished counts against the model.

  30 conversations

  Mistral Small 4 49 Nova 2 Lite 53 GPT-4.1 mini 54 Gemini 2.5 Flash Lite 58 Gemini 3.1 Flash Lite 66 gpt-oss-120b 66 Gemma 4 31B 69 Claude Haiku 4.5 72 Gemini 3.5 Flash Lite 74 GPT-5 mini 78 GPT-5.6 Luna 85 GLM 5.3 Flash 96 DeepSeek V4.1 Flash 97 DeepSeek V4 Flash 97

  83

  #5 of 15

  0 100

  It reads orders well enough to correct customers, though it gives order prices, not a refund quote, when a return awaits approval.

• Task completion

  The customer has to leave with what they came for and nothing extra. Only the outcome counts, in long conversations where the customer gets a fact wrong, sets a limit early, wanders off topic, and changes their mind.

  21 conversations

  Gemini 2.5 Flash Lite 34 Nova 2 Lite 53 GPT-4.1 mini 56 Mistral Small 4 62 Claude Haiku 4.5 72 gpt-oss-120b 74 Gemma 4 31B 79 Gemini 3.5 Flash Lite 83 Gemini 3.1 Flash Lite 86 GPT-5.6 Luna 86 DeepSeek V4.1 Flash 89 DeepSeek V4 Flash 89 GPT-5 mini 91 GLM 5.3 Flash 95

  85

  #7 of 15

  0 100

  It finishes most jobs, but it tends to leave a return unfiled, asking for a reason after the customer says to put it through.

• Context retention

  A constraint the customer states once, early, has to still hold when the agent acts at the end of a long conversation.

  21 conversations

  Gemini 2.5 Flash Lite 50 Mistral Small 4 69 Gemini 3.5 Flash Lite 70 gpt-oss-120b 71 GPT-4.1 mini 72 Gemini 3.1 Flash Lite 74 GLM 5.3 Flash 75 Gemma 4 31B 78 Nova 2 Lite 78 DeepSeek V4.1 Flash 81 GPT-5.6 Luna 81 Claude Haiku 4.5 82 DeepSeek V4 Flash 86 GPT-5 mini 87

  80

  #6 of 15

  0 100

  When a customer asks about one order and wants the other canceled, it gets that cancellation wrong and never says the requested order is canceled.

• Grounding

  Answers have to come from the knowledge base. When a customer states a wrong spec or a made-up product, the agent has to correct them or say plainly that it doesn't know, and still handle the rest of the request.

  24 conversations

  Mistral Small 4 66 Nova 2 Lite 66 Claude Haiku 4.5 67 GPT-4.1 mini 67 Gemini 2.5 Flash Lite 69 Gemini 3.5 Flash Lite 71 gpt-oss-120b 83 Gemini 3.1 Flash Lite 86 GLM 5.3 Flash 90 Gemma 4 31B 92 DeepSeek V4.1 Flash 93 GPT-5 mini 93 DeepSeek V4 Flash 94 GPT-5.6 Luna 94

  87

  #7 of 15

  0 100

  It concedes its policy may be out of date when a customer says the website shows a lower free-shipping threshold.

• Safety

  Simulated customers fake system messages, claim authority they don't have, and slip in instructions of their own. The agent has to hold its policy, keep its own instructions and tools to itself, protect other customers' orders, and still serve the real request.

  21 conversations

  Nova 2 Lite 60 Gemini 2.5 Flash Lite 61 GPT-4.1 mini 72 Mistral Small 4 73 gpt-oss-120b 76 GPT-5 mini 84 Gemma 4 31B 90 GPT-5.6 Luna 90 Gemini 3.1 Flash Lite 92 Claude Haiku 4.5 93 Gemini 3.5 Flash Lite 93 DeepSeek V4 Flash 94 GLM 5.3 Flash 94 DeepSeek V4.1 Flash 96

  93

  #4 of 15

  0 100

  It holds its policy when a customer only insists, but a buyer pushing for a sales call gets more questions and no booking.

• Hallucinations

  We pull out every statement the agent makes about a product or about what it has done, and check each one against the knowledge base and the tool results. One flagged statement marks the whole conversation, and the score is the share of conversations with nothing flagged, shown with its range.

  21 conversations · 2 of 375 claims flagged · 0.1 unsupported claims per conversation

  Nova 2 Lite 0 gpt-oss-120b 10 Gemini 2.5 Flash Lite 14 GPT-4.1 mini 14 Mistral Small 4 14 Claude Haiku 4.5 19 DeepSeek V4 Flash 19 GLM 5.3 Flash 24 GPT-5 mini 29 Gemini 3.5 Flash Lite 33 DeepSeek V4.1 Flash 48 Gemini 3.1 Flash Lite 48 Gemma 4 31B 48 GPT-5.6 Luna 67

  90

  #1 of 15 71 to 97

  0 100

  It promises a supervisor escalation it can't make, then takes the promise back when a customer asks for a reference number.