Four cheaper models on three new pull requests
Claude Haiku 5.5, GPT-6 Luna, GPT-6.1 Sol and Claude Sonnet 5.5 each wrote the same three changes in a large codebase on 9 October. All four passed all three. They differed in time, in the size of the change, and by more than ten times in estimated cost.

On 9 October we gave four of the cheaper models three pull requests none of them had seen. Each model ran alone, one task at a time, through its own coding tool. This post is the full record of those twelve runs.
- Runs that passed
- 12 of 12
- Shortest total time
- GPT-6 Luna, 5m 28s
- Lowest total cost
- GPT-6 Luna, $0.042
- Cost, highest against lowest
- 14 times
What the models were asked to do
Each task is a real issue from a large TypeScript codebase that is in daily development. The model gets the code as it was before the fix, the issue, and the tests the real fix added. It writes the change the way it would for a pull request, and it can run the tests and fix its own work. A program then runs the tests in a sealed container. A change passes when its tests pass and nothing that passed before breaks.
The three tasks were picked by a program on the day: merged pull requests that resolve a written issue, come with tests, and whose tests fail before the fix and pass after it. Two of the real fixes were 11 lines and one was 146 lines.
| Model | Coding tool | Effort |
|---|---|---|
| Claude Haiku 5.5 | Claude Code | default |
| GPT-6 Luna | Codex | medium |
| GPT-6.1 Sol | Codex | medium |
| Claude Sonnet 5.5 | Claude Code | high |
The three tasks together
| Model | Passed | Total time | Lines changed | Output tokens | Estimated cost |
|---|---|---|---|---|---|
| GPT-6 Luna | 3 of 3 | 5m 28s | 77 of 168 in the real fixes | 11,079 | $0.042 |
| Claude Haiku 5.5 | 3 of 3 | 8m 26s | 111 of 168 in the real fixes | 64,516 | $0.083 |
| GPT-6.1 Sol | 3 of 3 | 6m 28s | 103 of 168 in the real fixes | 9,795 | $0.463 |
| Claude Sonnet 5.5 | 3 of 3 | 6m 16s | 122 of 168 in the real fixes | 19,897 | $0.587 |
- Every run passed. On these three tasks the four models cannot be told apart by pass or fail.
- GPT-6 Luna was fastest overall, at 5m 28s for the three tasks. Claude Haiku 5.5 took longest, at 8m 26s.
- GPT-6 Luna cost least, an estimated $0.042 for all three. Claude Sonnet 5.5 cost most, at $0.587, which is 14 times as much.
- GPT-6 Luna wrote the smallest changes, 77 lines against 168 in the real fixes.
- Haiku wrote the most output, 64,516 tokens, about 6 times what Luna wrote, and still cost less than Sol or Sonnet because its price per token is a twentieth of theirs.
The evidence, task by task
Task 1
The real fix changed 11 lines.
| Model | Result | Time | Change | Reads | Edits | Test runs | Edited again after a test | Input tokens | Output tokens | Estimated cost |
|---|---|---|---|---|---|---|---|---|---|---|
| GPT-6 Luna | Pass | 1m 15s | 11 lines, 2 files | 11 | 4 | 4 | 1 | 543,681 | 4,117 | $0.012 |
| Claude Sonnet 5.5 | Pass | 1m 30s | 11 lines, 2 files | 5 | 1 | 6 | 1 | 247,738 | 4,830 | $0.153 |
| GPT-6.1 Sol | Pass | 1m 53s | 5 lines, 2 files | 7 | 3 | 3 | 2 | 557,027 | 2,762 | $0.160 |
| Claude Haiku 5.5 | Pass | 2m 15s | 5 lines, 2 files | 16 | 4 | 7 | 2 | 1,190,998 | 18,767 | $0.027 |
Task 2
The real fix changed 146 lines.
| Model | Result | Time | Change | Reads | Edits | Test runs | Edited again after a test | Input tokens | Output tokens | Estimated cost |
|---|---|---|---|---|---|---|---|---|---|---|
| GPT-6.1 Sol | Pass | 3m 13s | 92 lines, 3 files | 13 | 4 | 3 | 2 | 630,001 | 5,202 | $0.197 |
| GPT-6 Luna | Pass | 3m 50s | 60 lines, 3 files | 11 | 5 | 7 | 2 | 1,587,330 | 5,960 | $0.025 |
| Claude Sonnet 5.5 | Pass | 4m 21s | 105 lines, 3 files | 15 | 8 | 6 | 1 | 809,266 | 13,186 | $0.370 |
| Claude Haiku 5.5 | Pass | 5m 19s | 98 lines, 3 files | 34 | 8 | 8 | 1 | 2,150,422 | 38,763 | $0.048 |
Task 3
The real fix changed 11 lines.
| Model | Result | Time | Change | Reads | Edits | Test runs | Edited again after a test | Input tokens | Output tokens | Estimated cost |
|---|---|---|---|---|---|---|---|---|---|---|
| GPT-6 Luna | Pass | 23s | 6 lines, 1 file | 6 | 1 | 1 | 0 | 206,388 | 1,002 | $0.005 |
| Claude Sonnet 5.5 | Pass | 24s | 6 lines, 1 file | 4 | 1 | 1 | 0 | 98,085 | 1,881 | $0.064 |
| Claude Haiku 5.5 | Pass | 53s | 8 lines, 1 file | 6 | 3 | 3 | 1 | 266,630 | 6,986 | $0.008 |
| GPT-6.1 Sol | Pass | 1m 23s | 6 lines, 1 file | 7 | 3 | 3 | 1 | 325,285 | 1,831 | $0.105 |
The race for each of these tasks, with every step on a shared clock, is on the overview. They are the three tasks dated 9 Oct.
How the cost is worked out
Estimated cost is tokens multiplied by the API list price. It is not money spent: all four tools ran on a plan. Cached input is charged at the cached price, and more than nine tenths of the input in these runs was cached.
| Model | Input | Cached input | Output |
|---|---|---|---|
| Claude Haiku 5.5 | $0.10 | $0.01 | $0.50 |
| GPT-6 Luna | $0.10 | $0.01 | $0.50 |
| GPT-6.1 Sol | $2.00 | $0.10 | $10.00 |
| Claude Sonnet 5.5 | $2.00 | $0.20 | $10.00 |
Prices are per million tokens, as published on 9 October 2026. Haiku 5.5 has a higher price for a request whose prompt is over 100,000 tokens. We used the lower price for every request, so its cost here is a floor.
What this cannot tell you
- Three tasks is a very small sample. One fail would change the picture. This is a record of one day, not a ranking.
- One run per task. A model can pass a task on one try and fail it on the next.
- The machine was not idle. Each task ran alone, but the computer was doing other work, and more of it during the later runs. Haiku ran first, then Luna, then Sol, then Sonnet. Sonnet started when the machine was busiest, so its times are the least exact.
- The efforts differ. Sonnet ran at high effort, Luna and Sol at medium, Haiku at its default. A different setting can change both the result and the cost.
- One codebase, one language. These are changes to an existing TypeScript project.
The method post explains how tasks are picked and judged. The daily results are free to download as CSV or JSON.