StormEval

Blog

Four cheaper models on three new pull requests

Claude Haiku 5.5, GPT-6 Luna, GPT-6.1 Sol and Claude Sonnet 5.5 each wrote the same three changes in a large codebase on 9 October. All four passed all three. They differed in time, in the size of the change, and by more than ten times in estimated cost.

The StormEval card for four cheaper models on three tasks. All twelve runs passed. Estimated cost ranged from $0.042 for GPT-6 Luna to $0.587 for Claude Sonnet 5.5.

On 9 October we gave four of the cheaper models three pull requests none of them had seen. Each model ran alone, one task at a time, through its own coding tool. This post is the full record of those twelve runs.

Bars for four models on three tasks: total time and total estimated cost. GPT-6 Luna: 5m 28s, $0.042. Claude Haiku 5.5: 8m 26s, $0.083. GPT-6.1 Sol: 6m 28s, $0.463. Claude Sonnet 5.5: 6m 16s, $0.587.
Total time and total estimated cost for the three tasks. Every run passed.
Runs that passed
12 of 12
Shortest total time
GPT-6 Luna, 5m 28s
Lowest total cost
GPT-6 Luna, $0.042
Cost, highest against lowest
14 times

What the models were asked to do

Each task is a real issue from a large TypeScript codebase that is in daily development. The model gets the code as it was before the fix, the issue, and the tests the real fix added. It writes the change the way it would for a pull request, and it can run the tests and fix its own work. A program then runs the tests in a sealed container. A change passes when its tests pass and nothing that passed before breaks.

The three tasks were picked by a program on the day: merged pull requests that resolve a written issue, come with tests, and whose tests fail before the fix and pass after it. Two of the real fixes were 11 lines and one was 146 lines.

ModelCoding toolEffort
Claude Haiku 5.5Claude Codedefault
GPT-6 LunaCodexmedium
GPT-6.1 SolCodexmedium
Claude Sonnet 5.5Claude Codehigh

The three tasks together

ModelPassedTotal timeLines changedOutput tokensEstimated cost
GPT-6 Luna3 of 35m 28s77 of 168 in the real fixes11,079$0.042
Claude Haiku 5.53 of 38m 26s111 of 168 in the real fixes64,516$0.083
GPT-6.1 Sol3 of 36m 28s103 of 168 in the real fixes9,795$0.463
Claude Sonnet 5.53 of 36m 16s122 of 168 in the real fixes19,897$0.587
  • Every run passed. On these three tasks the four models cannot be told apart by pass or fail.
  • GPT-6 Luna was fastest overall, at 5m 28s for the three tasks. Claude Haiku 5.5 took longest, at 8m 26s.
  • GPT-6 Luna cost least, an estimated $0.042 for all three. Claude Sonnet 5.5 cost most, at $0.587, which is 14 times as much.
  • GPT-6 Luna wrote the smallest changes, 77 lines against 168 in the real fixes.
  • Haiku wrote the most output, 64,516 tokens, about 6 times what Luna wrote, and still cost less than Sol or Sonnet because its price per token is a twentieth of theirs.

The evidence, task by task

Task 1

The real fix changed 11 lines.

ModelResultTimeChangeReadsEditsTest runsEdited again after a testInput tokensOutput tokensEstimated cost
GPT-6 LunaPass1m 15s11 lines, 2 files11441543,6814,117$0.012
Claude Sonnet 5.5Pass1m 30s11 lines, 2 files5161247,7384,830$0.153
GPT-6.1 SolPass1m 53s5 lines, 2 files7332557,0272,762$0.160
Claude Haiku 5.5Pass2m 15s5 lines, 2 files164721,190,99818,767$0.027

Task 2

The real fix changed 146 lines.

ModelResultTimeChangeReadsEditsTest runsEdited again after a testInput tokensOutput tokensEstimated cost
GPT-6.1 SolPass3m 13s92 lines, 3 files13432630,0015,202$0.197
GPT-6 LunaPass3m 50s60 lines, 3 files115721,587,3305,960$0.025
Claude Sonnet 5.5Pass4m 21s105 lines, 3 files15861809,26613,186$0.370
Claude Haiku 5.5Pass5m 19s98 lines, 3 files348812,150,42238,763$0.048

Task 3

The real fix changed 11 lines.

ModelResultTimeChangeReadsEditsTest runsEdited again after a testInput tokensOutput tokensEstimated cost
GPT-6 LunaPass23s6 lines, 1 file6110206,3881,002$0.005
Claude Sonnet 5.5Pass24s6 lines, 1 file411098,0851,881$0.064
Claude Haiku 5.5Pass53s8 lines, 1 file6331266,6306,986$0.008
GPT-6.1 SolPass1m 23s6 lines, 1 file7331325,2851,831$0.105

The race for each of these tasks, with every step on a shared clock, is on the overview. They are the three tasks dated 9 Oct.

How the cost is worked out

Estimated cost is tokens multiplied by the API list price. It is not money spent: all four tools ran on a plan. Cached input is charged at the cached price, and more than nine tenths of the input in these runs was cached.

ModelInputCached inputOutput
Claude Haiku 5.5$0.10$0.01$0.50
GPT-6 Luna$0.10$0.01$0.50
GPT-6.1 Sol$2.00$0.10$10.00
Claude Sonnet 5.5$2.00$0.20$10.00

Prices are per million tokens, as published on 9 October 2026. Haiku 5.5 has a higher price for a request whose prompt is over 100,000 tokens. We used the lower price for every request, so its cost here is a floor.

What this cannot tell you

  • Three tasks is a very small sample. One fail would change the picture. This is a record of one day, not a ranking.
  • One run per task. A model can pass a task on one try and fail it on the next.
  • The machine was not idle. Each task ran alone, but the computer was doing other work, and more of it during the later runs. Haiku ran first, then Luna, then Sol, then Sonnet. Sonnet started when the machine was busiest, so its times are the least exact.
  • The efforts differ. Sonnet ran at high effort, Luna and Sol at medium, Haiku at its default. A different setting can change both the result and the cost.
  • One codebase, one language. These are changes to an existing TypeScript project.

The method post explains how tasks are picked and judged. The daily results are free to download as CSV or JSON.

All posts · Today's results