StormEval gives each model fresh issues in a working codebase and compares it with its own past. Each task's difficulty comes from how every model did on it. A model that gets worse under the same name shows up here, with a date.
Baselines are still building. No verdicts yet.
Models watched
10
With a verdict
0 of 10
Degraded now
–
Answers graded
145
Task raceOne task, every model, on one clock
Read
Edit
Command
Test run
Message
Finish (pass)
Finish (fail or out of time)
Each lane is one model working on the same task, starting at zero and ending where it finished. Every mark is one action it took.
4 Oct5 Oct8 Oct9 Oct15 Oct17 Oct21 Oct
Task 1, first run 4 Oct
Each model got the same issue in the same codebase. A change passes when the task's tests pass and nothing that passed before breaks. Real fix: 133 lines.
10 of 10 passed. Best on this task: GPT-6 Luna. Fastest pass: Claude Sonnet 5.5 in 1m 36s. Smallest passing change: GPT-6.1 Sol, 100 lines against 133 in the real fix.
1GPT-6 LunaCodex
Pass2m 23s13 reads · 5 edits · 5 tests4.9k out · 1.36M total · $0.024+116 lines · 3 files5 test runs, edited again 3 times
Passed in 2m 23s. 13 reads, 5 edits, 1 command, 5 test runs, 3 messages. Edited again after a test run 3 times. Changed 116 lines in 3 files; the real fix was 133 lines. Task Elo 1778. Elo 1778 after this task, up 278. Used 4.9k output tokens, 1.36M in all. Estimated cost $0.024 at list price.
2GPT-6.1 SolCodex
Pass2m 24sSmallest change14 reads · 4 edits · 3 tests6.1k out · 900k total · $0.280+100 lines · 3 files3 test runs, edited again once
Passed in 2m 24s. 14 reads, 4 edits, 1 command, 3 test runs, 5 messages. Edited again after a test run once. Changed 100 lines in 3 files; the real fix was 133 lines. Task Elo 1752. Elo 1752 after this task, up 252. Used 6.1k output tokens, 900k in all. Estimated cost $0.280 at list price.
3GPT-6 AstraCodex
Pass4m 24s21 reads · 3 edits · 4 tests5.8k out · 1.20M total · $2.02+113 lines · 4 files4 test runs, edited again 2 times
Passed in 4m 24s. 21 reads, 3 edits, 2 commands, 4 test runs, 5 messages. Edited again after a test run 2 times. Changed 113 lines in 4 files; the real fix was 133 lines. Task Elo 1596. Elo 1596 after this task, up 96. Used 5.8k output tokens, 1.20M in all. Estimated cost $2.02 at list price.
4Claude Sonnet 5.5Claude Code
Pass1m 36sFastest pass16 reads · 3 edits · 1 test9.7k out · 854k total · $0.385+120 lines · 4 files1 test run, edited again once
Passed in 1m 36s. 16 reads, 3 edits, 1 test run, 3 messages. Edited again after a test run once. Changed 120 lines in 4 files; the real fix was 133 lines. Task Elo 1557. Elo 1557 after this task, up 57. Used 9.7k output tokens, 854k in all. Estimated cost $0.385 at list price.
5Claude Haiku 5.5Claude Code
Pass9m 28s30 reads · 16 edits · 13 tests55k out · 5.11M total · $0.090+119 lines · 5 files13 test runs, edited again 2 times
Passed in 9m 28s. 30 reads, 16 edits, 7 commands, 13 test runs, 18 messages. Edited again after a test run 2 times. Changed 119 lines in 5 files; the real fix was 133 lines. Task Elo 1481. Elo 1481 after this task, down 19. Used 55k output tokens, 5.11M in all. Estimated cost $0.090 at list price.
6GPT-6 Luna (high)Codex
Pass3m 9s16 reads · 6 edits · 3 tests11k out · 2.12M total · $0.036+141 lines · 3 files3 test runs, edited again 2 times
Passed in 3m 9s. 16 reads, 6 edits, 2 commands, 3 test runs, 5 messages. Edited again after a test run 2 times. Changed 141 lines in 3 files; the real fix was 133 lines. Task Elo 1462. Elo 1462 after this task, down 38. Used 11k output tokens, 2.12M in all. Estimated cost $0.036 at list price.
6DeepSeek V4.1 FlashDeepSeek Harness
Pass7m 27s65 reads · 12 edits · 12 tests57k out · 7.94M total · $0.073+132 lines · 5 files12 test runs, edited again once
Passed in 7m 27s. 65 reads, 12 edits, 5 commands, 12 test runs, 1 message. Edited again after a test run once. Changed 132 lines in 5 files; the real fix was 133 lines. Task Elo 1462. Elo 1462 after this task, down 38. Used 57k output tokens, 7.94M in all. Estimated cost $0.073 at list price.
8Grok 4.7Cursor
Pass8m 32s79 reads · 10 edits · 1 test31k out · 3.75M total · $2.53+117 lines · 3 files1 test run
Passed in 8m 32s. 79 reads, 10 edits, 4 commands, 1 test run, 2 messages. Changed 117 lines in 3 files; the real fix was 133 lines. Task Elo 1443. Elo 1443 after this task, down 57. Used 31k output tokens, 3.75M in all. Estimated cost $2.53 at list price.
9Claude Opus 5.5Claude Code
Pass6m 26s7 reads · 5 edits · 6 tests17k out · 1.62M total · $1.34+157 lines · 6 files6 test runs, edited again 3 times
Passed in 6m 26s. 7 reads, 5 edits, 10 commands, 6 test runs, 5 messages. Edited again after a test run 3 times. Changed 157 lines in 6 files; the real fix was 133 lines. Task Elo 1274. Elo 1274 after this task, down 226. Used 17k output tokens, 1.62M in all. Estimated cost $1.34 at list price.
10Kimi K3Kimi Code
Pass11m 9s39 reads · 11 edits · 6 tests20k out · 4.04M total · $12.37+136 lines · 5 files6 test runs, edited again 2 times
Passed in 11m 9s. 39 reads, 11 edits, 4 commands, 6 test runs, 1 message. Edited again after a test run 2 times. Changed 136 lines in 5 files; the real fix was 133 lines. Task Elo 1192. Elo 1192 after this task, down 308. Used 20k output tokens, 4.04M in all. Estimated cost $12.37 at list price.
02m4m6m8m10m
Task 2, first run 4 Oct
Each model got the same issue in the same codebase. A change passes when the task's tests pass and nothing that passed before breaks. Real fix: 69 lines.
9 of 10 passed. Best on this task: GPT-6.1 Sol. Fastest pass: Claude Opus 5.5 in 3m 1s. Smallest passing change: GPT-6.1 Sol, 65 lines against 69 in the real fix.
1GPT-6.1 SolCodex
Pass3m 10sSmallest change23 reads · 1 edit · 3 tests5.4k out · 1.21M total · $0.280+65 lines · 4 files3 test runs
Passed in 3m 10s. 23 reads, 1 edit, 2 commands, 3 test runs, 6 messages. Changed 65 lines in 4 files; the real fix was 69 lines. Task Elo 1771. Elo 1765 after this task, up 13. Used 5.4k output tokens, 1.21M in all. Estimated cost $0.280 at list price.
2Claude Haiku 5.5Claude Code
Pass9m 7s37 reads · 11 edits · 18 tests51k out · 4.82M total · $0.084+66 lines · 4 files18 test runs, edited again once
Passed in 9m 7s. 37 reads, 11 edits, 10 commands, 18 test runs, 22 messages. Edited again after a test run once. Changed 66 lines in 4 files; the real fix was 69 lines. Task Elo 1634. Elo 1556 after this task, up 75. Used 51k output tokens, 4.82M in all. Estimated cost $0.084 at list price.
3GPT-6 AstraCodex
Pass4m 55s21 reads · 4 edits · 7 tests5.4k out · 1.25M total · $2.03+69 lines · 4 files7 test runs, edited again 2 times
Passed in 4m 55s. 21 reads, 4 edits, 3 commands, 7 test runs, 5 messages. Edited again after a test run 2 times. Changed 69 lines in 4 files; the real fix was 69 lines. Task Elo 1594. Elo 1594 after this task, down 2. Used 5.4k output tokens, 1.25M in all. Estimated cost $2.03 at list price.
4DeepSeek V4.1 FlashDeepSeek Harness
Pass5m 45s51 reads · 11 edits · 10 tests40k out · 5.35M total · $0.052+76 lines · 4 files10 test runs, edited again 2 times
Passed in 5m 45s. 51 reads, 11 edits, 9 commands, 10 test runs, 1 message. Edited again after a test run 2 times. Changed 76 lines in 4 files; the real fix was 69 lines. Task Elo 1575. Elo 1519 after this task, up 57. Used 40k output tokens, 5.35M in all. Estimated cost $0.052 at list price.
5Claude Opus 5.5Claude Code
Pass3m 1sFastest pass12 reads · 2 edits · 4 tests9.3k out · 590k total · $0.596+77 lines · 4 files4 test runs, edited again once
Passed in 3m 1s. 12 reads, 2 edits, 4 test runs, 3 messages. Edited again after a test run once. Changed 77 lines in 4 files; the real fix was 69 lines. Task Elo 1556. Elo 1425 after this task, up 151. Used 9.3k output tokens, 590k in all. Estimated cost $0.596 at list price.
6GPT-6 Luna (high)Codex
Pass4m 1s26 reads · 8 edits · 8 tests14k out · 2.99M total · $0.046+91 lines · 4 files8 test runs, edited again 3 times
Passed in 4m 1s. 26 reads, 8 edits, 3 commands, 8 test runs, 6 messages. Edited again after a test run 3 times. Changed 91 lines in 4 files; the real fix was 69 lines. Task Elo 1500. Elo 1482 after this task, up 20. Used 14k output tokens, 2.99M in all. Estimated cost $0.046 at list price.
7GPT-6 LunaCodex
Pass7m 17s28 reads · 10 edits · 7 tests13k out · 2.89M total · $0.048+85 lines · 4 files7 test runs, edited again 2 times
Passed in 7m 17s. 28 reads, 10 edits, 3 commands, 7 test runs, 7 messages. Edited again after a test run 2 times. Changed 85 lines in 4 files; the real fix was 69 lines. Task Elo 1462. Elo 1604 after this task, down 174. Used 13k output tokens, 2.89M in all. Estimated cost $0.048 at list price.
8Kimi K3Kimi Code
Pass9m 50s19 reads · 11 edits · 10 tests15k out · 5.19M total · $15.76+73 lines · 4 files10 test runs, edited again once
Passed in 9m 50s. 19 reads, 11 edits, 5 commands, 10 test runs, 1 message. Edited again after a test run once. Changed 73 lines in 4 files; the real fix was 69 lines. Task Elo 1423. Elo 1324 after this task, up 132. Used 15k output tokens, 5.19M in all. Estimated cost $15.76 at list price.
9Grok 4.7Cursor
Pass9m 47s73 reads · 11 edits · 5 tests37k out · 4.67M total · $3.10+81 lines · 4 files5 test runs, edited again once
Passed in 9m 47s. 73 reads, 11 edits, 4 commands, 5 test runs, 3 messages. Edited again after a test run once. Changed 81 lines in 4 files; the real fix was 69 lines. Task Elo 1339. Elo 1397 after this task, down 46. Used 37k output tokens, 4.67M in all. Estimated cost $3.10 at list price.
–Claude Sonnet 5.5Claude Code
Fail2m 8s12 reads · 13 edits · 4 tests12k out · 574k total · $0.308+74 lines · 4 files4 test runs, edited again 2 times
Failed in 2m 8s. 12 reads, 13 edits, 1 command, 4 test runs, 4 messages. Edited again after a test run 2 times. Changed 74 lines in 4 files; the real fix was 69 lines. Task Elo 993. Elo 1356 after this task, down 201. Used 12k output tokens, 574k in all. Estimated cost $0.308 at list price.
02m4m6m8m10m
Task 3, first run 4 Oct
Each model got the same issue in the same codebase. A change passes when the task's tests pass and nothing that passed before breaks. Real fix: 41 lines.
7 of 10 passed. Best on this task: GPT-6.1 Sol. Fastest pass: GPT-6.1 Sol in 1m 41s. Smallest passing change: Kimi K3, 27 lines against 41 in the real fix.
1GPT-6.1 SolCodex
Pass1m 41sFastest pass8 reads · 3 edits · 3 tests3.3k out · 536k total · $0.152+29 lines · 2 files3 test runs, edited again once
Passed in 1m 41s. 8 reads, 3 edits, 2 commands, 3 test runs, 5 messages. Edited again after a test run once. Changed 29 lines in 2 files; the real fix was 41 lines. Task Elo 1797. Elo 1772 after this task, up 7. Used 3.3k output tokens, 536k in all. Estimated cost $0.152 at list price.
2Claude Opus 5.5Claude Code
Pass1m 50s7 reads · 1 edit · 3 tests5.6k out · 324k total · $0.353+29 lines · 2 files3 test runs
Passed in 1m 50s. 7 reads, 1 edit, 3 test runs, 2 messages. Changed 29 lines in 2 files; the real fix was 41 lines. Task Elo 1720. Elo 1512 after this task, up 87. Used 5.6k output tokens, 324k in all. Estimated cost $0.353 at list price.
3Kimi K3Kimi Code
Pass6m 55sSmallest change14 reads · 6 edits · 6 tests9.0k out · 1.62M total · $4.96+27 lines · 2 files6 test runs, edited again once
Passed in 6m 55s. 14 reads, 6 edits, 2 commands, 6 test runs, 1 message. Edited again after a test run once. Changed 27 lines in 2 files; the real fix was 41 lines. Task Elo 1629. Elo 1426 after this task, up 102. Used 9.0k output tokens, 1.62M in all. Estimated cost $4.96 at list price.
4GPT-6 Luna (high)Codex
Pass1m 46s12 reads · 2 edits · 4 tests5.8k out · 924k total · $0.018+64 lines · 2 files4 test runs
Passed in 1m 46s. 12 reads, 2 edits, 2 commands, 4 test runs, 5 messages. Changed 64 lines in 2 files; the real fix was 41 lines. Task Elo 1607. Elo 1518 after this task, up 36. Used 5.8k output tokens, 924k in all. Estimated cost $0.018 at list price.
4GPT-6 AstraCodex
Pass2m 51s9 reads · 3 edits · 3 tests2.7k out · 429k total · $0.819+32 lines · 2 files3 test runs, edited again once
Passed in 2m 51s. 9 reads, 3 edits, 1 command, 3 test runs, 4 messages. Edited again after a test run once. Changed 32 lines in 2 files; the real fix was 41 lines. Task Elo 1607. Elo 1594 after this task, unchanged. Used 2.7k output tokens, 429k in all. Estimated cost $0.819 at list price.
6DeepSeek V4.1 FlashDeepSeek Harness
Pass3m 49s33 reads · 6 edits · 7 tests24k out · 2.57M total · $0.031+48 lines · 2 files7 test runs, edited again 2 times
Passed in 3m 49s. 33 reads, 6 edits, 4 commands, 7 test runs, 1 message. Edited again after a test run 2 times. Changed 48 lines in 2 files; the real fix was 41 lines. Task Elo 1586. Elo 1537 after this task, up 18. Used 24k output tokens, 2.57M in all. Estimated cost $0.031 at list price.
7Grok 4.7Cursor
Pass3m 44s35 reads · 9 edits · 3 tests15k out · 1.74M total · $1.09+61 lines · 2 files3 test runs, edited again once
Passed in 3m 44s. 35 reads, 9 edits, 1 command, 3 test runs, 3 messages. Edited again after a test run once. Changed 61 lines in 2 files; the real fix was 41 lines. Task Elo 1478. Elo 1426 after this task, up 29. Used 15k output tokens, 1.74M in all. Estimated cost $1.09 at list price.
–GPT-6 LunaCodex
Fail1m 39s8 reads · 2 edits · 1 test2.1k out · 333k total · $0.0094+20 lines · 2 files1 test run
Failed in 1m 39s. 8 reads, 2 edits, 1 command, 1 test run, 3 messages. Changed 20 lines in 2 files; the real fix was 41 lines. Task Elo 1123. Elo 1489 after this task, down 115. Used 2.1k output tokens, 333k in all. Estimated cost $0.0094 at list price.
–Claude Sonnet 5.5Claude Code
Out of time15m 0s6 reads · 1 edit · 4 tests– out · – total · –Change –4 test runs
Out of time at 15m 0s. 6 reads, 1 edit, 5 commands, 4 test runs, 3 messages. Task Elo 1123. Elo 1294 after this task, down 62.
–Claude Haiku 5.5Claude Code
Out of time15m 0s24 reads · 10 edits · 9 tests– out · – total · –Change –9 test runs, edited again 3 times
Out of time at 15m 0s. 24 reads, 10 edits, 4 commands, 9 test runs, 13 messages. Edited again after a test run 3 times. Task Elo 1123. Elo 1455 after this task, down 101.
05m10m15m
Task 4, first run 4 Oct
Each model got the same issue in the same codebase. A change passes when the task's tests pass and nothing that passed before breaks. Real fix: 77 lines.
10 of 10 passed. Best on this task: GPT-6.1 Sol and GPT-6 Luna. Fastest pass: GPT-6.1 Sol in 2m 22s. Smallest passing change: GPT-6 Astra, 44 lines against 77 in the real fix.
1GPT-6.1 SolCodex
Pass2m 22sFastest pass13 reads · 3 edits · 4 tests3.6k out · 944k total · $0.221+52 lines · 1 file4 test runs, edited again 2 times
Passed in 2m 22s. 13 reads, 3 edits, 2 commands, 4 test runs, 6 messages. Edited again after a test run 2 times. Changed 52 lines in 1 file; the real fix was 77 lines. Task Elo 1720. Elo 1764 after this task, down 8. Used 3.6k output tokens, 944k in all. Estimated cost $0.221 at list price.
1GPT-6 LunaCodex
Pass3m 48s9 reads · 2 edits · 5 tests4.8k out · 827k total · $0.019+65 lines · 1 file5 test runs
Passed in 3m 48s. 9 reads, 2 edits, 1 command, 5 test runs, 5 messages. Changed 65 lines in 1 file; the real fix was 77 lines. Task Elo 1720. Elo 1545 after this task, up 56. Used 4.8k output tokens, 827k in all. Estimated cost $0.019 at list price.
3GPT-6 AstraCodex
Pass4m 11sSmallest change18 reads · 1 edit · 4 tests3.5k out · 1.07M total · $1.77+44 lines · 1 file4 test runs
Passed in 4m 11s. 18 reads, 1 edit, 3 commands, 4 test runs, 5 messages. Changed 44 lines in 1 file; the real fix was 77 lines. Task Elo 1613. Elo 1601 after this task, up 7. Used 3.5k output tokens, 1.07M in all. Estimated cost $1.77 at list price.
4DeepSeek V4.1 FlashDeepSeek Harness
Pass5m 23s32 reads · 4 edits · 5 tests25k out · 2.44M total · $0.033+65 lines · 1 file5 test runs, edited again once
Passed in 5m 23s. 32 reads, 4 edits, 5 commands, 5 test runs, 1 message. Edited again after a test run once. Changed 65 lines in 1 file; the real fix was 77 lines. Task Elo 1575. Elo 1547 after this task, up 10. Used 25k output tokens, 2.44M in all. Estimated cost $0.033 at list price.
5Claude Haiku 5.5Claude Code
Pass3m 49s7 reads · 5 edits · 7 tests20k out · 929k total · $0.026+70 lines · 1 file7 test runs, edited again once
Passed in 3m 49s. 7 reads, 5 edits, 2 commands, 7 test runs, 6 messages. Edited again after a test run once. Changed 70 lines in 1 file; the real fix was 77 lines. Task Elo 1500. Elo 1464 after this task, up 9. Used 20k output tokens, 929k in all. Estimated cost $0.026 at list price.
6GPT-6 Luna (high)Codex
Pass3m 29s9 reads · 2 edits · 8 tests9.0k out · 2.16M total · $0.034+74 lines · 1 file8 test runs, edited again once
Passed in 3m 29s. 9 reads, 2 edits, 3 commands, 8 test runs, 6 messages. Edited again after a test run once. Changed 74 lines in 1 file; the real fix was 77 lines. Task Elo 1463. Elo 1505 after this task, down 13. Used 9.0k output tokens, 2.16M in all. Estimated cost $0.034 at list price.
6Claude Opus 5.5Claude Code
Pass4m 24s7 reads · 5 edits · 6 tests8.6k out · 490k total · $0.525+67 lines · 1 file6 test runs, edited again once
Passed in 4m 24s. 7 reads, 5 edits, 2 commands, 6 test runs, 3 messages. Edited again after a test run once. Changed 67 lines in 1 file; the real fix was 77 lines. Task Elo 1463. Elo 1500 after this task, down 12. Used 8.6k output tokens, 490k in all. Estimated cost $0.525 at list price.
8Claude Sonnet 5.5Claude Code
Pass4m 42s9 reads · 1 edit · 4 tests5.7k out · 354k total · $0.191+68 lines · 1 file4 test runs
Passed in 4m 42s. 9 reads, 1 edit, 4 test runs, 4 messages. Changed 68 lines in 1 file; the real fix was 77 lines. Task Elo 1425. Elo 1328 after this task, up 34. Used 5.7k output tokens, 354k in all. Estimated cost $0.191 at list price.
9Kimi K3Kimi Code
Pass13m 20s12 reads · 5 edits · 10 tests14k out · 2.02M total · $6.22+67 lines · 1 file10 test runs
Passed in 13m 20s. 12 reads, 5 edits, 2 commands, 10 test runs, 1 message. Changed 67 lines in 1 file; the real fix was 77 lines. Task Elo 1322. Elo 1400 after this task, down 26. Used 14k output tokens, 2.02M in all. Estimated cost $6.22 at list price.
10Grok 4.7Cursor
Pass6m 21s33 reads · 4 edits · 2 tests19k out · 1.37M total · $1.32+79 lines · 1 file2 test runs
Passed in 6m 21s. 33 reads, 4 edits, 1 command, 2 test runs, 4 messages. Changed 79 lines in 1 file; the real fix was 77 lines. Task Elo 1162. Elo 1370 after this task, down 56. Used 19k output tokens, 1.37M in all. Estimated cost $1.32 at list price.
05m10m
Task 5, first run 4 Oct
Each model got the same issue in the same codebase. A change passes when the task's tests pass and nothing that passed before breaks. Real fix: 156 lines.
8 of 9 passed. Best on this task: GPT-6.1 Sol. Fastest pass: GPT-6.1 Sol in 2m 30s. Smallest passing change: GPT-6.1 Sol and GPT-6 Astra, 98 lines against 156 in the real fix.
1GPT-6.1 SolCodex
Pass2m 30sFastest passSmallest change18 reads · 4 edits · 3 tests4.9k out · 1.28M total · $0.300+98 lines · 2 files3 test runs
Passed in 2m 30s. 18 reads, 4 edits, 2 commands, 3 test runs, 6 messages. Changed 98 lines in 2 files; the real fix was 156 lines. Task Elo 1744. Elo 1768 after this task, up 4. Used 4.9k output tokens, 1.28M in all. Estimated cost $0.300 at list price.
2GPT-6 AstraCodex
Pass3m 44sSmallest change18 reads · 3 edits · 4 tests4.5k out · 1.11M total · $1.89+98 lines · 2 files4 test runs, edited again once
Passed in 3m 44s. 18 reads, 3 edits, 3 commands, 4 test runs, 5 messages. Edited again after a test run once. Changed 98 lines in 2 files; the real fix was 156 lines. Task Elo 1646. Elo 1613 after this task, up 12. Used 4.5k output tokens, 1.11M in all. Estimated cost $1.89 at list price.
3GPT-6 Luna (high)Codex
Pass5m 25s30 reads · 11 edits · 9 tests19k out · 4.55M total · $0.068+188 lines · 2 files9 test runs, edited again 4 times
Passed in 5m 25s. 30 reads, 11 edits, 4 commands, 9 test runs, 6 messages. Edited again after a test run 4 times. Changed 188 lines in 2 files; the real fix was 156 lines. Task Elo 1541. Elo 1514 after this task, up 9. Used 19k output tokens, 4.55M in all. Estimated cost $0.068 at list price.
3GPT-6 LunaCodex
Pass12m 31s37 reads · 14 edits · 11 tests15k out · 5.34M total · $0.075+138 lines · 2 files11 test runs, edited again 6 times
Passed in 12m 31s. 37 reads, 14 edits, 7 commands, 11 test runs, 7 messages. Edited again after a test run 6 times. Changed 138 lines in 2 files; the real fix was 156 lines. Task Elo 1541. Elo 1546 after this task, up 1. Used 15k output tokens, 5.34M in all. Estimated cost $0.075 at list price.
5Claude Haiku 5.5Claude Code
Pass7m 28s43 reads · 11 edits · 9 tests79k out · 6.35M total · $0.117+145 lines · 3 files9 test runs, edited again 3 times
Passed in 7m 28s. 43 reads, 11 edits, 2 commands, 9 test runs, 38 messages. Edited again after a test run 3 times. Changed 145 lines in 3 files; the real fix was 156 lines. Task Elo 1520. Elo 1476 after this task, up 12. Used 79k output tokens, 6.35M in all. Estimated cost $0.117 at list price.
5Claude Sonnet 5.5Claude Code
Pass7m 53s22 reads · 3 edits · 9 tests27k out · 3.21M total · $1.12+126 lines · 3 files9 test runs, edited again 2 times
Passed in 7m 53s. 22 reads, 3 edits, 9 commands, 9 test runs, 6 messages. Edited again after a test run 2 times. Changed 126 lines in 3 files; the real fix was 156 lines. Task Elo 1520. Elo 1369 after this task, up 41. Used 27k output tokens, 3.21M in all. Estimated cost $1.12 at list price.
7Claude Opus 5.5Claude Code
Pass7m 2s15 reads · 2 edits · 5 tests18k out · 880k total · $1.02+170 lines · 2 files5 test runs, edited again once
Passed in 7m 2s. 15 reads, 2 edits, 5 test runs, 3 messages. Edited again after a test run once. Changed 170 lines in 2 files; the real fix was 156 lines. Task Elo 1459. Elo 1494 after this task, down 6. Used 18k output tokens, 880k in all. Estimated cost $1.02 at list price.
8DeepSeek V4.1 FlashDeepSeek Harness
Pass8m 46s84 reads · 9 edits · 9 tests68k out · 8.62M total · $0.082+234 lines · 3 files9 test runs, edited again 3 times
Passed in 8m 46s. 84 reads, 9 edits, 6 commands, 9 test runs, 1 message. Edited again after a test run 3 times. Changed 234 lines in 3 files; the real fix was 156 lines. Task Elo 1371. Elo 1518 after this task, down 29. Used 68k output tokens, 8.62M in all. Estimated cost $0.082 at list price.
–Grok 4.7Cursor
Out of time15m 0s63 reads · 6 edits · 1 test– out · – total · –Change –1 test run
Out of time at 15m 0s. 63 reads, 6 edits, 1 test run, 2 messages. Task Elo 1022. Elo 1328 after this task, down 42.
05m10m15m
Task 6, first run 4 Oct
Each model got the same issue in the same codebase. A change passes when the task's tests pass and nothing that passed before breaks. Real fix: 45 lines.
9 of 9 passed. Best on this task: GPT-6 Luna (high). Fastest pass: GPT-6.1 Sol in 2m 15s. Smallest passing change: GPT-6 Luna (high) and GPT-6 Luna, 27 lines against 45 in the real fix.
1GPT-6 Luna (high)Codex
Pass3m 43sSmallest change17 reads · 2 edits · 7 tests7.9k out · 1.08M total · $0.020+27 lines · 1 file7 test runs
Passed in 3m 43s. 17 reads, 2 edits, 1 command, 7 test runs, 6 messages. Changed 27 lines in 1 file; the real fix was 45 lines. Task Elo 1794. Elo 1553 after this task, up 39. Used 7.9k output tokens, 1.08M in all. Estimated cost $0.020 at list price.
2GPT-6 LunaCodex
Pass4m 12sSmallest change16 reads · 3 edits · 5 tests5.1k out · 1.39M total · $0.023+27 lines · 1 file5 test runs, edited again 2 times
Passed in 4m 12s. 16 reads, 3 edits, 7 commands, 5 test runs, 4 messages. Edited again after a test run 2 times. Changed 27 lines in 1 file; the real fix was 45 lines. Task Elo 1708. Elo 1571 after this task, up 25. Used 5.1k output tokens, 1.39M in all. Estimated cost $0.023 at list price.
3GPT-6.1 SolCodex
Pass2m 15sFastest pass17 reads · 1 edit · 5 tests3.9k out · 905k total · $0.226+31 lines · 1 file5 test runs, edited again once
Passed in 2m 15s. 17 reads, 1 edit, 2 commands, 5 test runs, 5 messages. Edited again after a test run once. Changed 31 lines in 1 file; the real fix was 45 lines. Task Elo 1683. Elo 1756 after this task, down 12. Used 3.9k output tokens, 905k in all. Estimated cost $0.226 at list price.
4GPT-6 AstraCodex
Pass3m 12s18 reads · 1 edit · 3 tests3.3k out · 885k total · $1.45+32 lines · 1 file3 test runs
Passed in 3m 12s. 18 reads, 1 edit, 1 command, 3 test runs, 4 messages. Changed 32 lines in 1 file; the real fix was 45 lines. Task Elo 1522. Elo 1600 after this task, down 13. Used 3.3k output tokens, 885k in all. Estimated cost $1.45 at list price.
5Claude Sonnet 5.5Claude Code
Pass2m 29s16 reads · 1 edit · 7 tests8.4k out · 821k total · $0.327+36 lines · 2 files7 test runs, edited again once
Passed in 2m 29s. 16 reads, 1 edit, 7 test runs, 8 messages. Edited again after a test run once. Changed 36 lines in 2 files; the real fix was 45 lines. Task Elo 1500. Elo 1392 after this task, up 23. Used 8.4k output tokens, 821k in all. Estimated cost $0.327 at list price.
6Claude Haiku 5.5Claude Code
Pass3m 57s14 reads · 6 edits · 11 tests20k out · 1.09M total · $0.026+37 lines · 1 file11 test runs, edited again 2 times
Passed in 3m 57s. 14 reads, 6 edits, 1 command, 11 test runs, 13 messages. Edited again after a test run 2 times. Changed 37 lines in 1 file; the real fix was 45 lines. Task Elo 1456. Elo 1474 after this task, down 2. Used 20k output tokens, 1.09M in all. Estimated cost $0.026 at list price.
7Claude Opus 5.5Claude Code
Pass5m 43s7 reads · 1 edit · 6 tests7.0k out · 423k total · $0.461+33 lines · 1 file6 test runs, edited again once
Passed in 5m 43s. 7 reads, 1 edit, 6 test runs, 3 messages. Edited again after a test run once. Changed 33 lines in 1 file; the real fix was 45 lines. Task Elo 1387. Elo 1481 after this task, down 13. Used 7.0k output tokens, 423k in all. Estimated cost $0.461 at list price.
8DeepSeek V4.1 FlashDeepSeek Harness
Pass5m 37s50 reads · 5 edits · 6 tests39k out · 4.84M total · $0.051+39 lines · 1 file6 test runs, edited again once
Passed in 5m 37s. 50 reads, 5 edits, 5 commands, 6 test runs, 1 message. Edited again after a test run once. Changed 39 lines in 1 file; the real fix was 45 lines. Task Elo 1337. Elo 1494 after this task, down 24. Used 39k output tokens, 4.84M in all. Estimated cost $0.051 at list price.
9Grok 4.7Cursor
Pass6m 41s62 reads · 6 edits · 4 tests23k out · 3.18M total · $1.89+62 lines · 1 file4 test runs, edited again once
Passed in 6m 41s. 62 reads, 6 edits, 4 test runs, 5 messages. Edited again after a test run once. Changed 62 lines in 1 file; the real fix was 45 lines. Task Elo 1004. Elo 1294 after this task, down 34. Used 23k output tokens, 3.18M in all. Estimated cost $1.89 at list price.
02m4m6m
Task 1, first run 5 Oct
Each model got the same issue in the same codebase. A change passes when the task's tests pass and nothing that passed before breaks. Real fix: 90 lines.
8 of 9 passed. Best on this task: GPT-6 Luna (high). Fastest pass: GPT-6.1 Sol in 1m 33s. Smallest passing change: GPT-6 Luna (high) and GPT-6 Astra, 34 lines against 90 in the real fix.
1GPT-6 Luna (high)Codex
Pass2m 2sSmallest change14 reads · 3 edits · 4 tests8.1k out · 1.00M total · $0.019+34 lines · 3 files4 test runs
Passed in 2m 2s. 14 reads, 3 edits, 2 commands, 4 test runs, 6 messages. Changed 34 lines in 3 files; the real fix was 90 lines. Task Elo 1884. Elo 1588 after this task, up 35. Used 8.1k output tokens, 1.00M in all. Estimated cost $0.019 at list price.
2GPT-6.1 SolCodex
Pass1m 33sFastest pass14 reads · 1 edit · 3 tests3.1k out · 572k total · $0.171+41 lines · 3 files3 test runs
Passed in 1m 33s. 14 reads, 1 edit, 2 commands, 3 test runs, 4 messages. Changed 41 lines in 3 files; the real fix was 90 lines. Task Elo 1720. Elo 1750 after this task, down 6. Used 3.1k output tokens, 572k in all. Estimated cost $0.171 at list price.
3GPT-6 AstraCodex
Pass2m 10sSmallest change9 reads · 3 edits · 5 tests3.5k out · 652k total · $1.60+34 lines · 3 files5 test runs, edited again once
Passed in 2m 10s. 9 reads, 3 edits, 2 commands, 5 test runs, 5 messages. Edited again after a test run once. Changed 34 lines in 3 files; the real fix was 90 lines. Task Elo 1642. Elo 1606 after this task, up 6. Used 3.5k output tokens, 652k in all. Estimated cost $1.60 at list price.
4DeepSeek V4.1 FlashDeepSeek Harness
Pass3m 38s49 reads · 5 edits · 7 tests37k out · 3.86M total · $0.043+54 lines · 3 files7 test runs, edited again 2 times
Passed in 3m 38s. 49 reads, 5 edits, 2 commands, 7 test runs, 1 message. Edited again after a test run 2 times. Changed 54 lines in 3 files; the real fix was 90 lines. Task Elo 1618. Elo 1510 after this task, up 16. Used 37k output tokens, 3.86M in all. Estimated cost $0.043 at list price.
5Claude Haiku 5.5Claude Code
Pass4m 1s16 reads · 4 edits · 13 tests29k out · 1.46M total · $0.035+84 lines · 3 files13 test runs, edited again 2 times
Passed in 4m 1s. 16 reads, 4 edits, 1 command, 13 test runs, 24 messages. Edited again after a test run 2 times. Changed 84 lines in 3 files; the real fix was 90 lines. Task Elo 1453. Elo 1474 after this task, unchanged. Used 29k output tokens, 1.46M in all. Estimated cost $0.035 at list price.
6Claude Sonnet 5.5Claude Code
Pass3m 56s10 reads · 7 edits · 4 tests9.4k out · 686k total · $0.316+73 lines · 3 files4 test runs
Passed in 3m 56s. 10 reads, 7 edits, 2 commands, 4 test runs, 4 messages. Changed 73 lines in 3 files; the real fix was 90 lines. Task Elo 1429. Elo 1401 after this task, up 9. Used 9.4k output tokens, 686k in all. Estimated cost $0.316 at list price.
6Grok 4.7Cursor
Pass9m 39s55 reads · 5 edits · 2 tests1.4k out · 416k total · $0.219+64 lines · 3 files2 test runs, edited again once
Passed in 9m 39s. 55 reads, 5 edits, 1 command, 2 test runs, 6 messages. Edited again after a test run once. Changed 64 lines in 3 files; the real fix was 90 lines. Task Elo 1429. Elo 1318 after this task, up 24. Used 1.4k output tokens, 416k in all. Estimated cost $0.219 at list price.
8Claude Opus 5.5Claude Code
Pass4m 21s10 reads · 6 edits · 7 tests9.0k out · 500k total · $0.508+86 lines · 3 files7 test runs, edited again once
Passed in 4m 21s. 10 reads, 6 edits, 7 test runs, 7 messages. Edited again after a test run once. Changed 86 lines in 3 files; the real fix was 90 lines. Task Elo 1239. Elo 1457 after this task, down 24. Used 9.0k output tokens, 500k in all. Estimated cost $0.508 at list price.
–GPT-6 LunaCodex
Fail25s7 reads · 1 edit · 1 test1.2k out · 231k total · $0.0052+3 lines · 1 file1 test run
Failed in 25s. 7 reads, 1 edit, 1 test run, 3 messages. Changed 3 lines in 1 file; the real fix was 90 lines. Task Elo 980. Elo 1516 after this task, down 55. Used 1.2k output tokens, 231k in all. Estimated cost $0.0052 at list price.
02m4m6m8m
Task 2, first run 5 Oct
Each model got the same issue in the same codebase. A change passes when the task's tests pass and nothing that passed before breaks. Real fix: 16 lines.
7 of 9 passed. Best on this task: Claude Haiku 5.5. Fastest pass: GPT-6 Astra in 1m 23s. Smallest passing change: Claude Haiku 5.5, GPT-6 Astra, GPT-6.1 Sol, Claude Opus 5.5 and DeepSeek V4.1 Flash, 16 lines against 16 in the real fix.
1Claude Haiku 5.5Claude Code
Pass1m 24sSmallest change12 reads · 7 edits · 6 tests12k out · 772k total · $0.018+16 lines · 4 files6 test runs, edited again once
Passed in 1m 24s. 12 reads, 7 edits, 3 commands, 6 test runs, 14 messages. Edited again after a test run once. Changed 16 lines in 4 files; the real fix was 16 lines. Task Elo 1735. Elo 1505 after this task, up 31. Used 12k output tokens, 772k in all. Estimated cost $0.018 at list price.
2GPT-6 AstraCodex
Pass1m 23sFastest passSmallest change8 reads · 2 edits · 2 tests1.7k out · 359k total · $0.695+16 lines · 4 files2 test runs, edited again once
Passed in 1m 23s. 8 reads, 2 edits, 2 test runs, 4 messages. Edited again after a test run once. Changed 16 lines in 4 files; the real fix was 16 lines. Task Elo 1660. Elo 1613 after this task, up 7. Used 1.7k output tokens, 359k in all. Estimated cost $0.695 at list price.
2GPT-6.1 SolCodex
Pass1m 34sSmallest change9 reads · 3 edits · 3 tests3.0k out · 441k total · $0.129+16 lines · 4 files3 test runs, edited again once
Passed in 1m 34s. 9 reads, 3 edits, 3 test runs, 5 messages. Edited again after a test run once. Changed 16 lines in 4 files; the real fix was 16 lines. Task Elo 1660. Elo 1739 after this task, down 11. Used 3.0k output tokens, 441k in all. Estimated cost $0.129 at list price.
4Claude Opus 5.5Claude Code
Pass1m 55sSmallest change8 reads · 9 edits · 6 tests5.1k out · 238k total · $0.265+16 lines · 4 files6 test runs, edited again once
Passed in 1m 55s. 8 reads, 9 edits, 6 test runs, 6 messages. Edited again after a test run once. Changed 16 lines in 4 files; the real fix was 16 lines. Task Elo 1613. Elo 1476 after this task, up 19. Used 5.1k output tokens, 238k in all. Estimated cost $0.265 at list price.
5DeepSeek V4.1 FlashDeepSeek Harness
Pass3m 10sSmallest change24 reads · 8 edits · 14 tests15k out · 1.69M total · $0.023+16 lines · 4 files14 test runs, edited again once
Passed in 3m 10s. 24 reads, 8 edits, 1 command, 14 test runs, 1 message. Edited again after a test run once. Changed 16 lines in 4 files; the real fix was 16 lines. Task Elo 1591. Elo 1520 after this task, up 10. Used 15k output tokens, 1.69M in all. Estimated cost $0.023 at list price.
6GPT-6 Luna (high)Codex
Pass2m 3s9 reads · 2 edits · 5 tests6.4k out · 661k total · $0.016+18 lines · 5 files5 test runs, edited again once
Passed in 2m 3s. 9 reads, 2 edits, 5 test runs, 5 messages. Edited again after a test run once. Changed 18 lines in 5 files; the real fix was 16 lines. Task Elo 1523. Elo 1581 after this task, down 7. Used 6.4k output tokens, 661k in all. Estimated cost $0.016 at list price.
7Grok 4.7Cursor
Pass3m 3s48 reads · 8 edits · 1 test9.5k out · 1.51M total · $1.11+18 lines · 5 files1 test run
Passed in 3m 3s. 48 reads, 8 edits, 1 test run, 3 messages. Changed 18 lines in 5 files; the real fix was 16 lines. Task Elo 1351. Elo 1326 after this task, up 8. Used 9.5k output tokens, 1.51M in all. Estimated cost $1.11 at list price.
–Claude Sonnet 5.5Claude Code
Fail1 test broke1m 14s4 reads · 10 edits · 6 tests4.8k out · 266k total · $0.142+16 lines · 4 files6 test runs, edited again once
Failed in 1m 14s. 4 reads, 10 edits, 6 test runs, 4 messages. Edited again after a test run once. Changed 16 lines in 4 files; the real fix was 16 lines. 1 test that passed before broke. Task Elo 1075. Elo 1375 after this task, down 26. Used 4.8k output tokens, 266k in all. Estimated cost $0.142 at list price.
–GPT-6 LunaCodex
Fail1m 42s7 reads · 1 edit · 4 tests4.5k out · 955k total · $0.018+21 lines · 4 files4 test runs
Failed in 1m 42s. 7 reads, 1 edit, 2 commands, 4 test runs, 4 messages. Changed 21 lines in 4 files; the real fix was 16 lines. Task Elo 1075. Elo 1481 after this task, down 35. Used 4.5k output tokens, 955k in all. Estimated cost $0.018 at list price.
01m2m3m
Task 3, first run 5 Oct
Each model got the same issue in the same codebase. A change passes when the task's tests pass and nothing that passed before breaks. Real fix: 131 lines.
9 of 9 passed. Best on this task: GPT-6 Luna. Fastest pass: Claude Sonnet 5.5 in 1m 47s. Smallest passing change: GPT-6 Luna, 83 lines against 131 in the real fix.
1GPT-6 LunaCodex
Pass3m 18sSmallest change14 reads · 6 edits · 3 tests4.9k out · 852k total · $0.018+83 lines · 3 files3 test runs, edited again 2 times
Passed in 3m 18s. 14 reads, 6 edits, 2 commands, 3 test runs, 4 messages. Edited again after a test run 2 times. Changed 83 lines in 3 files; the real fix was 131 lines. Task Elo 1823. Elo 1514 after this task, up 33. Used 4.9k output tokens, 852k in all. Estimated cost $0.018 at list price.
2GPT-6 Luna (high)Codex
Pass1m 56s14 reads · 5 edits · 5 tests7.9k out · 975k total · $0.019+142 lines · 3 files5 test runs, edited again once
Passed in 1m 56s. 14 reads, 5 edits, 2 commands, 5 test runs, 4 messages. Edited again after a test run once. Changed 142 lines in 3 files; the real fix was 131 lines. Task Elo 1680. Elo 1592 after this task, up 11. Used 7.9k output tokens, 975k in all. Estimated cost $0.019 at list price.
3GPT-6.1 SolCodex
Pass2m 29s11 reads · 4 edits · 4 tests5.7k out · 790k total · $0.236+101 lines · 3 files4 test runs, edited again once
Passed in 2m 29s. 11 reads, 4 edits, 2 commands, 4 test runs, 6 messages. Edited again after a test run once. Changed 101 lines in 3 files; the real fix was 131 lines. Task Elo 1632. Elo 1727 after this task, down 12. Used 5.7k output tokens, 790k in all. Estimated cost $0.236 at list price.
4GPT-6 AstraCodex
Pass5m 5s10 reads · 3 edits · 6 tests4.9k out · 655k total · $1.32+89 lines · 3 files6 test runs
Passed in 5m 5s. 10 reads, 3 edits, 4 commands, 6 test runs, 5 messages. Changed 89 lines in 3 files; the real fix was 131 lines. Task Elo 1543. Elo 1606 after this task, down 7. Used 4.9k output tokens, 655k in all. Estimated cost $1.32 at list price.
5Claude Sonnet 5.5Claude Code
Pass1m 47sFastest pass11 reads · 6 edits · 2 tests12k out · 609k total · $0.319+150 lines · 3 files2 test runs
Passed in 1m 47s. 11 reads, 6 edits, 2 test runs, 4 messages. Changed 150 lines in 3 files; the real fix was 131 lines. Task Elo 1521. Elo 1392 after this task, up 17. Used 12k output tokens, 609k in all. Estimated cost $0.319 at list price.
6Claude Opus 5.5Claude Code
Pass3m 37s13 reads · 3 edits · 3 tests10k out · 873k total · $0.737+147 lines · 3 files3 test runs
Passed in 3m 37s. 13 reads, 3 edits, 1 command, 3 test runs, 4 messages. Changed 147 lines in 3 files; the real fix was 131 lines. Task Elo 1457. Elo 1475 after this task, down 1. Used 10k output tokens, 873k in all. Estimated cost $0.737 at list price.
7Claude Haiku 5.5Claude Code
Pass6m 2s27 reads · 8 edits · 8 tests45k out · 2.81M total · $0.060+173 lines · 3 files8 test runs, edited again once
Passed in 6m 2s. 27 reads, 8 edits, 5 commands, 8 test runs, 25 messages. Edited again after a test run once. Changed 173 lines in 3 files; the real fix was 131 lines. Task Elo 1367. Elo 1492 after this task, down 13. Used 45k output tokens, 2.81M in all. Estimated cost $0.060 at list price.
8DeepSeek V4.1 FlashDeepSeek Harness
Pass5m 19s51 reads · 7 edits · 10 tests46k out · 5.17M total · $0.055+244 lines · 3 files10 test runs, edited again once
Passed in 5m 19s. 51 reads, 7 edits, 5 commands, 10 test runs, 1 message. Edited again after a test run once. Changed 244 lines in 3 files; the real fix was 131 lines. Task Elo 1318. Elo 1502 after this task, down 18. Used 46k output tokens, 5.17M in all. Estimated cost $0.055 at list price.
9Grok 4.7Cursor
Pass8m 13s69 reads · 8 edits · 2 tests31k out · 3.54M total · $2.69+200 lines · 3 files2 test runs
Passed in 8m 13s. 69 reads, 8 edits, 1 command, 2 test runs, 4 messages. Changed 200 lines in 3 files; the real fix was 131 lines. Task Elo 1133. Elo 1310 after this task, down 16. Used 31k output tokens, 3.54M in all. Estimated cost $2.69 at list price.
02m4m6m8m
Task 1, first run 8 Oct
Each model got the same issue in the same codebase. A change passes when the task's tests pass and nothing that passed before breaks. Real fix: 36 lines.
7 of 7 passed. Best on this task: GPT-6 Luna. Fastest pass: GPT-6 Luna in 40s. Smallest passing change: GPT-6.1 Sol and GPT-6 Astra, 14 lines against 36 in the real fix.
1GPT-6 LunaCodex
Pass40sFastest pass8 reads · 1 edit · 3 tests2.2k out · 369k total · $0.0083+16 lines · 1 file3 test runs
Passed in 40s. 8 reads, 1 edit, 1 command, 3 test runs, 4 messages. Changed 16 lines in 1 file; the real fix was 36 lines. Task Elo 1716. Elo 1532 after this task, up 18. Used 2.2k output tokens, 369k in all. Estimated cost $0.0083 at list price.
2GPT-6.1 SolCodex
Pass2m 4sSmallest change12 reads · 1 edit · 4 tests2.7k out · 601k total · $0.173+14 lines · 1 file4 test runs, edited again once
Passed in 2m 4s. 12 reads, 1 edit, 2 commands, 4 test runs, 5 messages. Edited again after a test run once. Changed 14 lines in 1 file; the real fix was 36 lines. Task Elo 1629. Elo 1722 after this task, down 5. Used 2.7k output tokens, 601k in all. Estimated cost $0.173 at list price.
3GPT-6 AstraCodex
Pass2m 9sSmallest change12 reads · 2 edits · 3 tests2.5k out · 669k total · $1.14+14 lines · 1 file3 test runs, edited again once
Passed in 2m 9s. 12 reads, 2 edits, 1 command, 3 test runs, 5 messages. Edited again after a test run once. Changed 14 lines in 1 file; the real fix was 36 lines. Task Elo 1550. Elo 1604 after this task, down 2. Used 2.5k output tokens, 669k in all. Estimated cost $1.14 at list price.
4Claude Sonnet 5.5Claude Code
Pass56s12 reads · 1 edit · 2 tests4.8k out · 349k total · $0.171+19 lines · 1 file2 test runs
Passed in 56s. 12 reads, 1 edit, 2 test runs, 2 messages. Changed 19 lines in 1 file; the real fix was 36 lines. Task Elo 1525. Elo 1407 after this task, up 15. Used 4.8k output tokens, 349k in all. Estimated cost $0.171 at list price.
5Claude Haiku 5.5Claude Code
Pass3m 18s20 reads · 3 edits · 7 tests15k out · 703k total · $0.018+19 lines · 1 file7 test runs, edited again once
Passed in 3m 18s. 20 reads, 3 edits, 2 commands, 7 test runs, 16 messages. Edited again after a test run once. Changed 19 lines in 1 file; the real fix was 36 lines. Task Elo 1500. Elo 1495 after this task, up 3. Used 15k output tokens, 703k in all. Estimated cost $0.018 at list price.
6Claude Opus 5.5Claude Code
Pass3m 21s7 reads · 1 edit · 6 tests5.1k out · 349k total · $0.334+22 lines · 1 file6 test runs, edited again once
Passed in 3m 21s. 7 reads, 1 edit, 6 test runs, 5 messages. Edited again after a test run once. Changed 22 lines in 1 file; the real fix was 36 lines. Task Elo 1279. Elo 1464 after this task, down 11. Used 5.1k output tokens, 349k in all. Estimated cost $0.334 at list price.
6GPT-6 Luna (high)Codex
Pass3m 25s16 reads · 2 edits · 6 tests8.6k out · 2.01M total · $0.032+31 lines · 1 file6 test runs, edited again once
Passed in 3m 25s. 16 reads, 2 edits, 7 commands, 6 test runs, 6 messages. Edited again after a test run once. Changed 31 lines in 1 file; the real fix was 36 lines. Task Elo 1279. Elo 1570 after this task, down 22. Used 8.6k output tokens, 2.01M in all. Estimated cost $0.032 at list price.
01m2m3m
Task 2, first run 8 Oct
Each model got the same issue in the same codebase. A change passes when the task's tests pass and nothing that passed before breaks. Real fix: 52 lines.
7 of 7 passed. Best on this task: GPT-6 Luna. Fastest pass: GPT-6 Luna in 53s. Smallest passing change: GPT-6 Astra, 31 lines against 52 in the real fix.
1GPT-6 LunaCodex
Pass53sFastest pass4 reads · 1 edit · 2 tests2.5k out · 370k total · $0.0077+37 lines · 1 file2 test runs
Passed in 53s. 4 reads, 1 edit, 3 commands, 2 test runs, 4 messages. Changed 37 lines in 1 file; the real fix was 52 lines. Task Elo 1746. Elo 1550 after this task, up 18. Used 2.5k output tokens, 370k in all. Estimated cost $0.0077 at list price.
2Claude Opus 5.5Claude Code
Pass1m 46s6 reads · 1 edit · 4 tests5.0k out · 285k total · $0.319+37 lines · 1 file4 test runs
Passed in 1m 46s. 6 reads, 1 edit, 1 command, 4 test runs, 3 messages. Changed 37 lines in 1 file; the real fix was 52 lines. Task Elo 1549. Elo 1473 after this task, up 9. Used 5.0k output tokens, 285k in all. Estimated cost $0.319 at list price.
2GPT-6 Luna (high)Codex
Pass1m 54s7 reads · 1 edit · 5 tests6.0k out · 489k total · $0.012+38 lines · 1 file5 test runs
Passed in 1m 54s. 7 reads, 1 edit, 5 commands, 5 test runs, 5 messages. Changed 38 lines in 1 file; the real fix was 52 lines. Task Elo 1549. Elo 1571 after this task, up 1. Used 6.0k output tokens, 489k in all. Estimated cost $0.012 at list price.
2GPT-6 AstraCodex
Pass2m 0sSmallest change8 reads · 3 edits · 5 tests2.9k out · 487k total · $0.932+31 lines · 1 file5 test runs, edited again once
Passed in 2m 0s. 8 reads, 3 edits, 2 commands, 5 test runs, 5 messages. Edited again after a test run once. Changed 31 lines in 1 file; the real fix was 52 lines. Task Elo 1549. Elo 1603 after this task, down 1. Used 2.9k output tokens, 487k in all. Estimated cost $0.932 at list price.
5Claude Sonnet 5.5Claude Code
Pass1m 42s6 reads · 1 edit · 4 tests4.2k out · 271k total · $0.144+48 lines · 2 files4 test runs
Passed in 1m 42s. 6 reads, 1 edit, 4 test runs, 3 messages. Changed 48 lines in 2 files; the real fix was 52 lines. Task Elo 1451. Elo 1413 after this task, up 6. Used 4.2k output tokens, 271k in all. Estimated cost $0.144 at list price.
6GPT-6.1 SolCodex
Pass2m 2s7 reads · 3 edits · 5 tests3.1k out · 498k total · $0.203+39 lines · 1 file5 test runs, edited again once
Passed in 2m 2s. 7 reads, 3 edits, 1 command, 5 test runs, 5 messages. Edited again after a test run once. Changed 39 lines in 1 file; the real fix was 52 lines. Task Elo 1375. Elo 1692 after this task, down 30. Used 3.1k output tokens, 498k in all. Estimated cost $0.203 at list price.
7Claude Haiku 5.5Claude Code
Pass2m 27s8 reads · 5 edits · 9 tests19k out · 802k total · $0.022+56 lines · 2 files9 test runs
Passed in 2m 27s. 8 reads, 5 edits, 3 commands, 9 test runs, 18 messages. Changed 56 lines in 2 files; the real fix was 52 lines. Task Elo 1288. Elo 1483 after this task, down 12. Used 19k output tokens, 802k in all. Estimated cost $0.022 at list price.
030s1m1m 30s2m2m 30s
Task 3, first run 8 Oct
Each model got the same issue in the same codebase. A change passes when the task's tests pass and nothing that passed before breaks. Real fix: 246 lines.
7 of 7 passed. Best on this task: GPT-6 Luna. Fastest pass: GPT-6 Luna in 2m 13s. Smallest passing change: GPT-6 Luna, 145 lines against 246 in the real fix.
1GPT-6 LunaCodex
Pass2m 13sFastest passSmallest change15 reads · 9 edits · 6 tests8.2k out · 1.12M total · $0.022+145 lines · 3 files6 test runs, edited again 3 times
Passed in 2m 13s. 15 reads, 9 edits, 3 commands, 6 test runs, 7 messages. Edited again after a test run 3 times. Changed 145 lines in 3 files; the real fix was 246 lines. Task Elo 1940. Elo 1572 after this task, up 22. Used 8.2k output tokens, 1.12M in all. Estimated cost $0.022 at list price.
2GPT-6 Luna (high)Codex
Pass2m 24s10 reads · 8 edits · 8 tests9.8k out · 1.70M total · $0.030+149 lines · 4 files8 test runs, edited again 2 times
Passed in 2m 24s. 10 reads, 8 edits, 2 commands, 8 test runs, 5 messages. Edited again after a test run 2 times. Changed 149 lines in 4 files; the real fix was 246 lines. Task Elo 1677. Elo 1580 after this task, up 9. Used 9.8k output tokens, 1.70M in all. Estimated cost $0.030 at list price.
3GPT-6.1 SolCodex
Pass3m 28s8 reads · 5 edits · 3 tests6.4k out · 717k total · $0.234+146 lines · 4 files3 test runs, edited again once
Passed in 3m 28s. 8 reads, 5 edits, 4 commands, 3 test runs, 6 messages. Edited again after a test run once. Changed 146 lines in 4 files; the real fix was 246 lines. Task Elo 1556. Elo 1685 after this task, down 7. Used 6.4k output tokens, 717k in all. Estimated cost $0.234 at list price.
4GPT-6 AstraCodex
Pass2m 45s8 reads · 3 edits · 2 tests4.9k out · 608k total · $1.55+151 lines · 4 files2 test runs, edited again once
Passed in 2m 45s. 8 reads, 3 edits, 2 commands, 2 test runs, 5 messages. Edited again after a test run once. Changed 151 lines in 4 files; the real fix was 246 lines. Task Elo 1445. Elo 1595 after this task, down 8. Used 4.9k output tokens, 608k in all. Estimated cost $1.55 at list price.
5Claude Sonnet 5.5Claude Code
Pass3m 1s6 reads · 3 edits · 5 tests12k out · 552k total · $0.314+236 lines · 5 files5 test runs, edited again once
Passed in 3m 1s. 6 reads, 3 edits, 5 test runs, 4 messages. Edited again after a test run once. Changed 236 lines in 5 files; the real fix was 246 lines. Task Elo 1418. Elo 1416 after this task, up 3. Used 12k output tokens, 552k in all. Estimated cost $0.314 at list price.
6Claude Haiku 5.5Claude Code
Pass3m 16s11 reads · 13 edits · 9 tests39k out · 1.65M total · $0.044+241 lines · 5 files9 test runs, edited again once
Passed in 3m 16s. 11 reads, 13 edits, 2 commands, 9 test runs, 19 messages. Edited again after a test run once. Changed 241 lines in 5 files; the real fix was 246 lines. Task Elo 1332. Elo 1475 after this task, down 8. Used 39k output tokens, 1.65M in all. Estimated cost $0.044 at list price.
7Claude Opus 5.5Claude Code
Pass3m 28s2 reads · 3 edits · 6 tests12k out · 434k total · $0.552+237 lines · 5 files6 test runs, edited again 2 times
Passed in 3m 28s. 2 reads, 3 edits, 6 test runs, 3 messages. Edited again after a test run 2 times. Changed 237 lines in 5 files; the real fix was 246 lines. Task Elo 1234. Elo 1462 after this task, down 11. Used 12k output tokens, 434k in all. Estimated cost $0.552 at list price.
01m2m3m
Task 1, first run 9 Oct
Each model got the same issue in the same codebase. A change passes when the task's tests pass and nothing that passed before breaks. Real fix: 11 lines.
4 of 4 passed. Best on this task: GPT-6 Luna and Claude Haiku 5.5. Fastest pass: GPT-6 Luna in 1m 15s. Smallest passing change: Claude Haiku 5.5 and GPT-6.1 Sol, 5 lines against 11 in the real fix.
1GPT-6 LunaCodex
Pass1m 15sFastest pass11 reads · 4 edits · 4 tests4.1k out · 548k total · $0.012+11 lines · 2 files4 test runs, edited again once
Passed in 1m 15s. 11 reads, 4 edits, 2 commands, 4 test runs, 4 messages. Edited again after a test run once. Changed 11 lines in 2 files; the real fix was 11 lines. Task Elo 1535. Elo 1572 after this task, unchanged. Used 4.1k output tokens, 548k in all. Estimated cost $0.012 at list price.
1Claude Haiku 5.5Claude Code
Pass2m 15sSmallest change16 reads · 4 edits · 7 tests19k out · 1.21M total · $0.027+5 lines · 2 files7 test runs, edited again 2 times
Passed in 2m 15s. 16 reads, 4 edits, 2 commands, 7 test runs, 16 messages. Edited again after a test run 2 times. Changed 5 lines in 2 files; the real fix was 11 lines. Task Elo 1535. Elo 1479 after this task, up 4. Used 19k output tokens, 1.21M in all. Estimated cost $0.027 at list price.
3GPT-6.1 SolCodex
Pass1m 53sSmallest change7 reads · 3 edits · 3 tests2.8k out · 560k total · $0.160+5 lines · 2 files3 test runs, edited again 2 times
Passed in 1m 53s. 7 reads, 3 edits, 4 commands, 3 test runs, 5 messages. Edited again after a test run 2 times. Changed 5 lines in 2 files; the real fix was 11 lines. Task Elo 1500. Elo 1678 after this task, down 7. Used 2.8k output tokens, 560k in all. Estimated cost $0.160 at list price.
4Claude Sonnet 5.5Claude Code
Pass1m 30s5 reads · 1 edit · 6 tests4.8k out · 253k total · $0.153+11 lines · 2 files6 test runs, edited again once
Passed in 1m 30s. 5 reads, 1 edit, 6 test runs, 3 messages. Edited again after a test run once. Changed 11 lines in 2 files; the real fix was 11 lines. Task Elo 1429. Elo 1417 after this task, up 1. Used 4.8k output tokens, 253k in all. Estimated cost $0.153 at list price.
030s1m1m 30s2m
Task 2, first run 9 Oct
Each model got the same issue in the same codebase. A change passes when the task's tests pass and nothing that passed before breaks. Real fix: 146 lines.
4 of 4 passed. Best on this task: GPT-6 Luna. Fastest pass: GPT-6.1 Sol in 3m 13s. Smallest passing change: GPT-6 Luna, 60 lines against 146 in the real fix.
1GPT-6 LunaCodex
Pass3m 50sSmallest change11 reads · 5 edits · 7 tests6.0k out · 1.59M total · $0.025+60 lines · 3 files7 test runs, edited again 2 times
Passed in 3m 50s. 11 reads, 5 edits, 2 commands, 7 test runs, 6 messages. Edited again after a test run 2 times. Changed 60 lines in 3 files; the real fix was 146 lines. Task Elo 1721. Elo 1579 after this task, up 7. Used 6.0k output tokens, 1.59M in all. Estimated cost $0.025 at list price.
2GPT-6.1 SolCodex
Pass3m 13sFastest pass13 reads · 4 edits · 3 tests5.2k out · 635k total · $0.197+92 lines · 3 files3 test runs, edited again 2 times
Passed in 3m 13s. 13 reads, 4 edits, 2 commands, 3 test runs, 6 messages. Edited again after a test run 2 times. Changed 92 lines in 3 files; the real fix was 146 lines. Task Elo 1583. Elo 1677 after this task, down 1. Used 5.2k output tokens, 635k in all. Estimated cost $0.197 at list price.
3Claude Haiku 5.5Claude Code
Pass5m 19s34 reads · 8 edits · 8 tests39k out · 2.19M total · $0.048+98 lines · 3 files8 test runs, edited again once
Passed in 5m 19s. 34 reads, 8 edits, 3 commands, 8 test runs, 24 messages. Edited again after a test run once. Changed 98 lines in 3 files; the real fix was 146 lines. Task Elo 1417. Elo 1478 after this task, down 1. Used 39k output tokens, 2.19M in all. Estimated cost $0.048 at list price.
4Claude Sonnet 5.5Claude Code
Pass4m 21s15 reads · 8 edits · 6 tests13k out · 822k total · $0.370+105 lines · 3 files6 test runs, edited again once
Passed in 4m 21s. 15 reads, 8 edits, 2 commands, 6 test runs, 6 messages. Edited again after a test run once. Changed 105 lines in 3 files; the real fix was 146 lines. Task Elo 1279. Elo 1412 after this task, down 5. Used 13k output tokens, 822k in all. Estimated cost $0.370 at list price.
01m2m3m4m5m
Task 3, first run 9 Oct
Each model got the same issue in the same codebase. A change passes when the task's tests pass and nothing that passed before breaks. Real fix: 11 lines.
4 of 4 passed. Best on this task: GPT-6 Luna. Fastest pass: GPT-6 Luna in 23s. Smallest passing change: GPT-6 Luna, Claude Sonnet 5.5 and GPT-6.1 Sol, 6 lines against 11 in the real fix.
1GPT-6 LunaCodex
Pass23sFastest passSmallest change6 reads · 1 edit · 1 test1.0k out · 207k total · $0.0046+6 lines · 1 file1 test run
Passed in 23s. 6 reads, 1 edit, 1 test run, 3 messages. Changed 6 lines in 1 file; the real fix was 11 lines. Task Elo 1658. Elo 1584 after this task, up 5. Used 1.0k output tokens, 207k in all. Estimated cost $0.0046 at list price.
2Claude Sonnet 5.5Claude Code
Pass24sSmallest change4 reads · 1 edit · 1 test1.9k out · 100k total · $0.064+6 lines · 1 file1 test run
Passed in 24s. 4 reads, 1 edit, 1 test run, 2 messages. Changed 6 lines in 1 file; the real fix was 11 lines. Task Elo 1537. Elo 1418 after this task, up 6. Used 1.9k output tokens, 100k in all. Estimated cost $0.064 at list price.
3GPT-6.1 SolCodex
Pass1m 23sSmallest change7 reads · 3 edits · 3 tests1.8k out · 327k total · $0.105+6 lines · 1 file3 test runs, edited again once
Passed in 1m 23s. 7 reads, 3 edits, 2 commands, 3 test runs, 4 messages. Edited again after a test run once. Changed 6 lines in 1 file; the real fix was 11 lines. Task Elo 1425. Elo 1667 after this task, down 10. Used 1.8k output tokens, 327k in all. Estimated cost $0.105 at list price.
4Claude Haiku 5.5Claude Code
Pass53s6 reads · 3 edits · 3 tests7.0k out · 274k total · $0.0084+8 lines · 1 file3 test runs, edited again once
Passed in 53s. 6 reads, 3 edits, 1 command, 3 test runs, 10 messages. Edited again after a test run once. Changed 8 lines in 1 file; the real fix was 11 lines. Task Elo 1385. Elo 1475 after this task, down 3. Used 7.0k output tokens, 274k in all. Estimated cost $0.0084 at list price.
015s30s45s1m1m 15s
Task 1, first run 15 Oct
Each model got the same issue in the same codebase. A change passes when the task's tests pass and nothing that passed before breaks. Real fix: 17 lines.
4 of 4 passed. Best on this task: GPT-6.1 Sol. Fastest pass: Claude Sonnet 5.5 in 33s. Smallest passing change: GPT-6.1 Sol, GPT-6 Astra and Claude Opus 5.5, 13 lines against 17 in the real fix.
1GPT-6.1 SolCodex
Pass2m 13sSmallest change11 reads · 1 edit · 3 tests2.1k out · 482k total · $0.128+13 lines · 1 file3 test runs
Passed in 2m 13s. 11 reads, 1 edit, 1 command, 3 test runs, 4 messages. Changed 13 lines in 1 file; the real fix was 17 lines. Task Elo 1571. Elo 1665 after this task, down 2. Used 2.1k output tokens, 482k in all. Estimated cost $0.128 at list price.
2Claude Sonnet 5.5Claude Code
Pass33sFastest pass6 reads · 1 edit · 1 test2.9k out · 150k total · $0.105+18 lines · 1 file1 test run
Passed in 33s. 6 reads, 1 edit, 1 test run, 2 messages. Changed 18 lines in 1 file; the real fix was 17 lines. Task Elo 1500. Elo 1423 after this task, up 5. Used 2.9k output tokens, 150k in all. Estimated cost $0.105 at list price.
3GPT-6 AstraCodex
Pass2m 15sSmallest change10 reads · 1 edit · 3 tests2.2k out · 450k total · $0.942+13 lines · 1 file3 test runs
Passed in 2m 15s. 10 reads, 1 edit, 1 command, 3 test runs, 5 messages. Changed 13 lines in 1 file; the real fix was 17 lines. Task Elo 1465. Elo 1590 after this task, down 4. Used 2.2k output tokens, 450k in all. Estimated cost $0.942 at list price.
3Claude Opus 5.5Claude Code
Pass2m 59sSmallest change9 reads · 2 edits · 5 tests3.5k out · 280k total · $0.275+13 lines · 1 file5 test runs, edited again once
Passed in 2m 59s. 9 reads, 2 edits, 5 test runs, 4 messages. Edited again after a test run once. Changed 13 lines in 1 file; the real fix was 17 lines. Task Elo 1465. Elo 1463 after this task, up 1. Used 3.5k output tokens, 280k in all. Estimated cost $0.275 at list price.
01m2m3m
Task 2, first run 15 Oct
Each model got the same issue in the same codebase. A change passes when the task's tests pass and nothing that passed before breaks. Real fix: 64 lines.
4 of 4 passed. Best on this task: GPT-6.1 Sol. Fastest pass: Claude Sonnet 5.5 in 1m 28s. Smallest passing change: GPT-6.1 Sol and GPT-6 Astra, 37 lines against 64 in the real fix.
1GPT-6.1 SolCodex
Pass1m 55sSmallest change9 reads · 1 edit · 3 tests2.8k out · 654k total · $0.199+37 lines · 2 files3 test runs
Passed in 1m 55s. 9 reads, 1 edit, 3 commands, 3 test runs, 4 messages. Changed 37 lines in 2 files; the real fix was 64 lines. Task Elo 1612. Elo 1666 after this task, up 1. Used 2.8k output tokens, 654k in all. Estimated cost $0.199 at list price.
2Claude Sonnet 5.5Claude Code
Pass1m 28sFastest pass6 reads · 4 edits · 5 tests7.3k out · 277k total · $0.170+63 lines · 2 files5 test runs
Passed in 1m 28s. 6 reads, 4 edits, 5 test runs, 3 messages. Changed 63 lines in 2 files; the real fix was 64 lines. Task Elo 1500. Elo 1426 after this task, up 3. Used 7.3k output tokens, 277k in all. Estimated cost $0.170 at list price.
2GPT-6 AstraCodex
Pass2m 8sSmallest change10 reads · 1 edit · 3 tests2.4k out · 421k total · $0.849+37 lines · 2 files3 test runs
Passed in 2m 8s. 10 reads, 1 edit, 1 command, 3 test runs, 4 messages. Changed 37 lines in 2 files; the real fix was 64 lines. Task Elo 1500. Elo 1588 after this task, down 2. Used 2.4k output tokens, 421k in all. Estimated cost $0.849 at list price.
4Claude Opus 5.5Claude Code
Pass3m 10s10 reads · 3 edits · 4 tests5.4k out · 361k total · $0.343+51 lines · 2 files4 test runs
Passed in 3m 10s. 10 reads, 3 edits, 4 test runs, 5 messages. Changed 51 lines in 2 files; the real fix was 64 lines. Task Elo 1388. Elo 1460 after this task, down 3. Used 5.4k output tokens, 361k in all. Estimated cost $0.343 at list price.
01m2m3m
Task 3, first run 15 Oct
Each model got the same issue in the same codebase. A change passes when the task's tests pass and nothing that passed before breaks. Real fix: 72 lines.
4 of 4 passed. Best on this task: GPT-6.1 Sol. Fastest pass: Claude Sonnet 5.5 in 1m 30s. Smallest passing change: GPT-6 Astra, 37 lines against 72 in the real fix.
1GPT-6.1 SolCodex
Pass2m 14s12 reads · 1 edit · 3 tests3.6k out · 748k total · $0.206+40 lines · 1 file3 test runs
Passed in 2m 14s. 12 reads, 1 edit, 3 commands, 3 test runs, 4 messages. Changed 40 lines in 1 file; the real fix was 72 lines. Task Elo 1612. Elo 1666 after this task, unchanged. Used 3.6k output tokens, 748k in all. Estimated cost $0.206 at list price.
2GPT-6 AstraCodex
Pass3m 0sSmallest change14 reads · 1 edit · 2 tests3.0k out · 657k total · $1.16+37 lines · 1 file2 test runs
Passed in 3m 0s. 14 reads, 1 edit, 1 command, 2 test runs, 6 messages. Changed 37 lines in 1 file; the real fix was 72 lines. Task Elo 1536. Elo 1588 after this task, unchanged. Used 3.0k output tokens, 657k in all. Estimated cost $1.16 at list price.
3Claude Sonnet 5.5Claude Code
Pass1m 30sFastest pass6 reads · 2 edits · 3 tests11k out · 322k total · $0.237+52 lines · 1 file3 test runs, edited again once
Passed in 1m 30s. 6 reads, 2 edits, 1 command, 3 test runs, 4 messages. Edited again after a test run once. Changed 52 lines in 1 file; the real fix was 72 lines. Task Elo 1464. Elo 1428 after this task, up 2. Used 11k output tokens, 322k in all. Estimated cost $0.237 at list price.
4Claude Opus 5.5Claude Code
Pass4m 6s7 reads · 1 edit · 7 tests7.3k out · 570k total · $0.539+44 lines · 1 file7 test runs, edited again once
Passed in 4m 6s. 7 reads, 1 edit, 7 test runs, 5 messages. Edited again after a test run once. Changed 44 lines in 1 file; the real fix was 72 lines. Task Elo 1388. Elo 1458 after this task, down 2. Used 7.3k output tokens, 570k in all. Estimated cost $0.539 at list price.
01m2m3m4m
Task 1, first run 17 Oct
Each model got the same issue in the same codebase. A change passes when the task's tests pass and nothing that passed before breaks. Real fix: 33 lines.
3 of 3 passed. Best on this task: Claude Sonnet 5.5. Fastest pass: Claude Sonnet 5.5 in 1m 2s. Smallest passing change: Claude Sonnet 5.5, 22 lines against 33 in the real fix.
1Claude Sonnet 5.5Claude Code
Pass1m 2sFastest passSmallest change2 reads · 1 edit · 6 tests3.9k out · 197k total · $0.122+22 lines · 1 file6 test runs
Passed in 1m 2s. 2 reads, 1 edit, 6 test runs, 2 messages. Changed 22 lines in 1 file; the real fix was 33 lines. Task Elo 1649. Elo 1434 after this task, up 6. Used 3.9k output tokens, 197k in all. Estimated cost $0.122 at list price.
2Claude Haiku 5.5Claude Code
Pass1m 42s4 reads · 1 edit · 9 tests9.6k out · 374k total · $0.012+23 lines · 1 file9 test runs, edited again once
Passed in 1m 42s. 4 reads, 1 edit, 9 test runs, 10 messages. Edited again after a test run once. Changed 23 lines in 1 file; the real fix was 33 lines. Task Elo 1500. Elo 1475 after this task, unchanged. Used 9.6k output tokens, 374k in all. Estimated cost $0.012 at list price.
3Claude Opus 5.5Claude Code
Pass2m 5s6 reads · 1 edit · 5 tests4.8k out · 412k total · $0.358+23 lines · 1 file5 test runs
Passed in 2m 5s. 6 reads, 1 edit, 5 commands, 5 test runs, 4 messages. Changed 23 lines in 1 file; the real fix was 33 lines. Task Elo 1351. Elo 1453 after this task, down 5. Used 4.8k output tokens, 412k in all. Estimated cost $0.358 at list price.
030s1m1m 30s2m
Task 2, first run 17 Oct
Each model got the same issue in the same codebase. A change passes when the task's tests pass and nothing that passed before breaks. Real fix: 69 lines.
3 of 3 passed. Best on this task: Claude Sonnet 5.5 and Claude Opus 5.5. Fastest pass: Claude Sonnet 5.5 in 1m 18s. Smallest passing change: Claude Opus 5.5, 53 lines against 69 in the real fix.
1Claude Sonnet 5.5Claude Code
Pass1m 18sFastest pass4 reads · 2 edits · 4 tests4.7k out · 214k total · $0.124+56 lines · 1 file4 test runs, edited again once
Passed in 1m 18s. 4 reads, 2 edits, 4 test runs, 3 messages. Edited again after a test run once. Changed 56 lines in 1 file; the real fix was 69 lines. Task Elo 1545. Elo 1436 after this task, up 2. Used 4.7k output tokens, 214k in all. Estimated cost $0.124 at list price.
1Claude Opus 5.5Claude Code
Pass1m 53sSmallest change4 reads · 2 edits · 6 tests4.1k out · 267k total · $0.276+53 lines · 1 file6 test runs, edited again once
Passed in 1m 53s. 4 reads, 2 edits, 6 test runs, 5 messages. Edited again after a test run once. Changed 53 lines in 1 file; the real fix was 69 lines. Task Elo 1545. Elo 1455 after this task, up 2. Used 4.1k output tokens, 267k in all. Estimated cost $0.276 at list price.
3Claude Haiku 5.5Claude Code
Pass2m 29s9 reads · 1 edit · 8 tests17k out · 662k total · $0.019+59 lines · 1 file8 test runs
Passed in 2m 29s. 9 reads, 1 edit, 1 command, 8 test runs, 16 messages. Changed 59 lines in 1 file; the real fix was 69 lines. Task Elo 1409. Elo 1471 after this task, down 4. Used 17k output tokens, 662k in all. Estimated cost $0.019 at list price.
030s1m1m 30s2m2m 30s
Task 3, first run 17 Oct
Each model got the same issue in the same codebase. A change passes when the task's tests pass and nothing that passed before breaks. Real fix: 48 lines.
3 of 3 passed. Best on this task: Claude Opus 5.5. Fastest pass: Claude Sonnet 5.5 in 2m 23s. Smallest passing change: Claude Opus 5.5, 32 lines against 48 in the real fix.
1Claude Opus 5.5Claude Code
Pass2m 44sSmallest change4 reads · 1 edit · 8 tests4.8k out · 389k total · $0.377+32 lines · 3 files8 test runs, edited again once
Passed in 2m 44s. 4 reads, 1 edit, 8 test runs, 5 messages. Edited again after a test run once. Changed 32 lines in 3 files; the real fix was 48 lines. Task Elo 1544. Elo 1456 after this task, up 1. Used 4.8k output tokens, 389k in all. Estimated cost $0.377 at list price.
2Claude Sonnet 5.5Claude Code
Pass2m 23sFastest pass7 reads · 2 edits · 8 tests6.4k out · 459k total · $0.212+48 lines · 3 files8 test runs, edited again once
Passed in 2m 23s. 7 reads, 2 edits, 8 test runs, 5 messages. Edited again after a test run once. Changed 48 lines in 3 files; the real fix was 48 lines. Task Elo 1500. Elo 1436 after this task, unchanged. Used 6.4k output tokens, 459k in all. Estimated cost $0.212 at list price.
3Claude Haiku 5.5Claude Code
Pass3m 8s18 reads · 12 edits · 8 tests22k out · 1.89M total · $0.036+48 lines · 3 files8 test runs, edited again once
Passed in 3m 8s. 18 reads, 12 edits, 10 commands, 8 test runs, 25 messages. Edited again after a test run once. Changed 48 lines in 3 files; the real fix was 48 lines. Task Elo 1456. Elo 1469 after this task, down 2. Used 22k output tokens, 1.89M in all. Estimated cost $0.036 at list price.
01m2m3m
Task 1, first run 21 Oct
Each model got the same issue in the same codebase. A change passes when the task's tests pass and nothing that passed before breaks. Real fix: 124 lines.
2 of 2 passed. Best on this task: Claude Sonnet 5.5 and Claude Opus 5.5. Fastest pass: Claude Sonnet 5.5 in 3m 27s. Smallest passing change: Claude Opus 5.5, 135 lines against 124 in the real fix.
1Claude Sonnet 5.5Claude Code
Pass3m 27sFastest pass9 reads · 11 edits · 6 tests18k out · 1.13M total · $0.524+218 lines · 3 files6 test runs, edited again once
Passed in 3m 27s. 9 reads, 11 edits, 1 command, 6 test runs, 7 messages. Edited again after a test run once. Changed 218 lines in 3 files; the real fix was 124 lines. Task Elo 1500. Elo 1436 after this task, unchanged. Used 18k output tokens, 1.13M in all. Estimated cost $0.524 at list price.
1Claude Opus 5.5Claude Code
Pass4m 55sSmallest change11 reads · 4 edits · 6 tests18k out · 1.11M total · $1.02+135 lines · 3 files6 test runs, edited again once
Passed in 4m 55s. 11 reads, 4 edits, 3 commands, 6 test runs, 11 messages. Edited again after a test run once. Changed 135 lines in 3 files; the real fix was 124 lines. Task Elo 1500. Elo 1456 after this task, unchanged. Used 18k output tokens, 1.11M in all. Estimated cost $1.02 at list price.
01m2m3m4m5m
Task 2, first run 21 Oct
Each model got the same issue in the same codebase. A change passes when the task's tests pass and nothing that passed before breaks. Real fix: 63 lines.
2 of 2 passed. Best on this task: Claude Opus 5.5. Fastest pass: Claude Opus 5.5 in 3m 38s. Smallest passing change: Claude Opus 5.5, 42 lines against 63 in the real fix.
1Claude Opus 5.5Claude Code
Pass3m 38sFastest passSmallest change15 reads · 9 edits · 9 tests12k out · 1.06M total · $0.869+42 lines · 2 files9 test runs, edited again 2 times
Passed in 3m 38s. 15 reads, 9 edits, 2 commands, 9 test runs, 8 messages. Edited again after a test run 2 times. Changed 42 lines in 2 files; the real fix was 63 lines. Task Elo 1560. Elo 1457 after this task, up 1. Used 12k output tokens, 1.06M in all. Estimated cost $0.869 at list price.
2Claude Sonnet 5.5Claude Code
Pass4m 35s17 reads · 19 edits · 12 tests18k out · 1.77M total · $0.660+48 lines · 2 files12 test runs, edited again 2 times
Passed in 4m 35s. 17 reads, 19 edits, 3 commands, 12 test runs, 9 messages. Edited again after a test run 2 times. Changed 48 lines in 2 files; the real fix was 63 lines. Task Elo 1440. Elo 1435 after this task, down 1. Used 18k output tokens, 1.77M in all. Estimated cost $0.660 at list price.
01m2m3m4m
Task 3, first run 21 Oct
Each model got the same issue in the same codebase. A change passes when the task's tests pass and nothing that passed before breaks. Real fix: 92 lines.
1 of 2 passed. Best on this task: Claude Opus 5.5.
1Claude Opus 5.5Claude Code
Pass3m 31s14 reads · 6 edits · 7 tests11k out · 915k total · $0.777+96 lines · 3 files7 test runs, edited again 2 times
Passed in 3m 31s. 14 reads, 6 edits, 1 command, 7 test runs, 8 messages. Edited again after a test run 2 times. Changed 96 lines in 3 files; the real fix was 92 lines. Task Elo 1631. Elo 1460 after this task, up 3. Used 11k output tokens, 915k in all. Estimated cost $0.777 at list price.
–Claude Sonnet 5.5Claude Code
Fail1 test broke1m 28s18 reads · 1 edit · 2 tests8.9k out · 973k total · $0.383+87 lines · 3 files2 test runs
Failed in 1m 28s. 18 reads, 1 edit, 1 command, 2 test runs, 4 messages. Changed 87 lines in 3 files; the real fix was 92 lines. 1 test that passed before broke. Task Elo 1369. Elo 1433 after this task, down 2. Used 8.9k output tokens, 973k in all. Estimated cost $0.383 at list price.
01m2m3m
Estimated cost is tokens at API list prices, not money spent: the Claude and Codex runs use plan usage. A dash means the run stored no token count.
01 · ShareTake the chart with you
One card with every model, its rating and its status, sized for a post or a slide. Save it as an image or as a short video.
Not a leaderboard. Models run at different effort settings and through different tools.
Tasks graded
1453 timeouts · 34 not scored
Time spent
9h 16mover 13 days
Tokens used
198.9M197.2M in · 1.78M out
Estimated cost
$97.55at API list prices
Estimated cost is tokens at API list prices, not money spent: the Claude and Codex runs use plan usage. A dash means no list price is set for that model. Graded counts use passing and failing runs; time, tokens and cost can also include finished runs excluded from grading.
Claude Sonnet 5.5median 1m 41sslowest timed out · pull-request
Kimi K3median 10m 30sslowest 13m 20s · pull-request
under 15s
15–30s
30–60s
1–2 min
2–4 min
over 4 min
timed out
By task family
Pass rate, mean time and mean output tokens for each kind of problem.
Model
pull-request
GPT-6 Luna (high)
100%2m 56s9.5k
Claude Opus 5.5
100%3m 34s8.7k
Claude Haiku 5.5
94%4m 42s29k
DeepSeek V4.1 Flash
100%5m 26s39k
Grok 4.7
89%7m 40s21k
GPT-6 Astra
100%3m 05s3.5k
GPT-6.1 Sol
100%2m 15s3.9k
GPT-6 Luna
80%3m 06s5.3k
Claude Sonnet 5.5
83%2m 54s9.2k
Kimi K3
100%10m 19s15k
Day by day
GPT-6 Luna (high)
1 day · time spent per day
Day
Passed
Timeouts
Time
Est. cost
8 Oct
12/12
0
35m 17s
$0.352
Claude Opus 5.5
7 days · time spent per day
Day
Passed
Timeouts
Time
Est. cost
21 Oct
3/3
0
12m 00s
$2.67
17 Oct
3/3
0
6m 38s
$1.01
15 Oct
3/3
0
10m 10s
$1.16
8 Oct
3/3
0
8m 31s
$1.21
7 Oct
3/3
0
9m 47s
$1.51
5 Oct
3/3
0
16m 35s
$2.01
4 Oct
3/3
0
11m 14s
$2.29
Claude Haiku 5.5
3 days · time spent per day
Day
Passed
Timeouts
Time
Est. cost
17 Oct
3/3
0
7m 15s
$0.067
9 Oct
3/3
0
8m 21s
$0.083
8 Oct
11/12
1
1h 09m
$0.540
DeepSeek V4.1 Flash
2 days · time spent per day
Day
Passed
Timeouts
Time
Est. cost
7 Oct
3/3
0
12m 07s
$0.121
5 Oct
6/6
0
36m 47s
$0.322
Grok 4.7
3 days · time spent per day
Day
Passed
Timeouts
Time
Est. cost
7 Oct
3/3
0
19m 33s
$4.02
5 Oct
2/3
1
27m 21s
$3.22
4 Oct
3/3
0
22m 04s
$6.72
GPT-6 Astra
5 days · time spent per day
Day
Passed
Timeouts
Time
Est. cost
15 Oct
3/3
0
7m 23s
$2.95
8 Oct
3/3
0
6m 54s
$3.62
7 Oct
3/3
0
8m 37s
$3.62
5 Oct
3/3
0
11m 07s
$5.11
4 Oct
3/3
0
12m 10s
$4.87
GPT-6.1 Sol
4 days · time spent per day
Day
Passed
Timeouts
Time
Est. cost
15 Oct
3/3
0
6m 22s
$0.533
9 Oct
3/3
0
6m 28s
$0.463
8 Oct
3/3
0
7m 34s
$0.611
7 Oct
9/9
0
19m 58s
$2.00
GPT-6 Luna
3 days · time spent per day
Day
Passed
Timeouts
Time
Est. cost
9 Oct
3/3
0
5m 28s
$0.042
8 Oct
3/3
0
3m 46s
$0.038
7 Oct
6/9
0
37m 15s
$0.238
Claude Sonnet 5.5
7 days · time spent per day
Day
Passed
Timeouts
Time
Est. cost
21 Oct
2/3
0
9m 25s
$1.57
17 Oct
3/3
0
4m 39s
$0.458
15 Oct
3/3
0
3m 26s
$0.512
9 Oct
3/3
0
6m 11s
$0.587
8 Oct
3/3
0
5m 36s
$0.628
7 Oct
2/3
0
6m 46s
$0.777
4 Oct
4/6
1
33m 41s
$2.33
Kimi K3
Last 7 days · 3 graded days total · time spent per day
Day
Passed
Timeouts
Time
Est. cost
14 Oct
0/0
0
0.0s
–
13 Oct
0/0
0
0.0s
–
12 Oct
0/0
0
0.0s
–
11 Oct
0/0
0
0.0s
–
10 Oct
0/0
0
0.0s
–
9 Oct
1/1
0
13m 20s
$6.22
8 Oct
2/2
0
16m 45s
$20.72
04 · MethodHow it works
Fresh tasks every day.Each task is an issue in a large, working codebase. The model has to write the change that resolves it, the way it would for a pull request. Every model is sent each task, and only one run per task counts toward a model's results. A model that has not run a task yet shows no result for it.
Done with the model's own coding tool.The model works in a copy of the code through its coding tool, the way a developer uses it. It can read the code, run the tests and fix its own work. Only the final change counts.
Graded by a program.The tests for the change must pass, and the tests that passed before must still pass. No model judges another model.
Rated like a chess player.A change that passes counts as a win against that task's difficulty, which comes from how every model did on it. The rating comes with an error band. Movement inside the band means nothing.
Compared with itself.The last 7 days are tested against the model's first 14 days.
Two signals.Degraded means the pass rate fell. Changed means the model spends a different number of tokens on the work while the pass rate held. Watch is an early warning: the pass rate dipped, but not enough to call.
How each number is computed
Rating
Fitted from passes and fails against each task's difficulty, like a chess rating. The ± is one standard error. The ± on a change combines the errors of the two ratings it compares, so a change inside it means nothing.
Elo
Every pair of models on a task plays one game. A passing change beats a failing one. When both pass, the game is split over the size of the change (counts double), time and estimated cost, using the measures both runs have. Task Elo is fitted on that task alone, and rows are ordered by it. Elo starts at 1500 and is fitted over every task so far. It is separate from the rating, which compares each model with its own past.
Pass rate
Passes divided by passes plus fails. The model table and its detail card show the latest graded day; the report shows all days. A change passes only when its tests pass and nothing that worked before breaks. Errors and refusals are left out.
Timeout
The change was not finished within 15 minutes. It is graded as a fail and counted as 15 minutes of time spent.
Time per task
Seconds from sending the task to the coding tool finishing. The mean and the median cover every task that finished or timed out.
Tokens
Input, output and thinking tokens as the provider reports them, for answered tasks. Thinking share is thinking tokens over output tokens.
Output tokens per second
A task's output tokens divided by its time, for answered tasks only. The table shows the median and the mean.
Estimated cost
Tokens multiplied by the API list price per million. It is not money spent: the Claude and Codex runs use plan usage. A model with no list price shows a dash.
Task race
Each lane starts when the coding tool was started. A mark sits at the moment the tool reported that step. Runs graded before step times were stored show only their start, their finish and their step count.
Tasks
Issues from a large codebase that is in daily development, each one a change a developer would open a pull request for. Every model gets the same set.
The daily results behind every number are free to download as CSV or JSON: one row per model per day. The tasks, the tests and the code the models wrote are not included.