StormEval

Blog

Claude Haiku 5.5 against GPT-6 Luna on 12 real pull requests

Both small models at high effort, given the same 12 issues in a large codebase and checked by the same tests. Luna passed all 12, faster and with fewer tokens. Haiku passed 11, and its changes were closer in size to the real fixes.

The StormEval head-to-head card. Claude Haiku 5.5 at high effort passed 11 of 12 tasks with a median of 3m 58s; GPT-6 Luna at high effort passed 12 of 12 with a median of 2m 46s.

Claude Haiku 5.5 came out this week, so we put it next to GPT-6 Luna, a small model from OpenAI. Both ran at high effort, each through its own coding tool, on the same 12 tasks the other models on this site are given.

Bars for seven models: tasks passed out of 12 and median time per task. GPT-6 Luna at high effort passed 12 in a median of 2m 46s. Claude Haiku 5.5 at high effort passed 11 in a median of 3m 58s.
Passed out of 12 and median time per task, for every model that has run all 12 tasks.

What the models were asked to do

Each task is a real issue from a large TypeScript codebase that is in daily development. The model gets the code as it was before the fix, the issue, and the tests the real fix added. It has to write the change, the way it would for a pull request, and it can run the tests and fix its own work. A program then runs the tests in a sealed container. A change passes when its tests pass and nothing that passed before breaks.

A run that is not finished in 15 minutes is a fail.

The head to head

GPT-6 Luna (high) passed
12 of 12
Claude Haiku 5.5 (high) passed
11 of 12
Luna, median time per task
2m 46s
Haiku, median time per task
3m 58s
Claude Haiku 5.5 (high)GPT-6 Luna (high)
Passed11 of 1212 of 12
Median time per task3m 58s2m 46s
Total time, all 12 tasks69 min35 min
Median output tokens per task28,6878,334
Size of passing changes, against the real fixes0.98 times0.91 times
  • Luna was the more reliable of the two. It passed every task, took about half as long in total, and wrote about a third of the output tokens.
  • Haiku's one fail was a timeout. On one task it was still working when the 15 minutes ran out. Claude Sonnet 5.5 and GPT-6 Luna at medium effort also failed that task; the other models that ran it passed, most in under five minutes.
  • Haiku's changes were closer in size to the real fixes. Across its passing tasks it wrote 0.98 lines for every line of the real fix. Luna wrote 0.91.

Every model on the same 12 tasks

ModelEffortPassedMedian timeMedian output tokensChange size against the real fix
Claude Opus 5.5medium12 of 123m 31s8,7560.99
GPT-6.1 Solmedium12 of 122m 18s3,7440.67
GPT-6 Astramedium12 of 123m 02s3,4940.66
GPT-6 Lunahigh12 of 122m 46s8,3340.91
Claude Haiku 5.5high11 of 123m 58s28,6870.98
Claude Sonnet 5.5high9 of 122m 17s9,3750.91
GPT-6 Lunamedium9 of 122m 18s4,8340.75

Two things stand out beyond the head to head. Luna at high effort passed the three tasks it failed at medium effort, so for this model the effort setting changed the result. And GPT-6.1 Sol and GPT-6 Astra write changes about two thirds the size of the real fixes, while the Claude models and Luna at high effort come closer to the real size.

DeepSeek V4.1 Flash and Grok 4.7 have run 9 of these tasks and Kimi K3 one, so they are left out of this table. Their results are on the overview.

What this cannot tell you

  • Twelve tasks is a small sample. One more fail would move a model from first place to the middle. Treat the order as a first look, not a ranking.
  • One graded run per task. A model can pass a task on one try and fail it on the next.
  • The times are not perfectly even. Haiku and Luna at high effort ran while the machine was also doing other work, so their times are less exact than the others'. Each task still ran alone, one at a time.
  • One codebase, one language. These are changes to an existing TypeScript project. They are not new projects, front-end design or long conversations.
  • It is a snapshot, not a verdict. StormEval's job is to show when a model gets worse than it was. That needs weeks of data; this is the first week for Haiku 5.5 and for Luna at high effort.

The method post explains how tasks are picked and judged. The daily results are free to download as CSV or JSON.

All posts · Today's results