StormEval

Updated 2026-10-21 04:41 UTC

Same name. Fresh tasks. Every day.

Is your AI model getting worse?

StormEval gives each model fresh issues in a working codebase and compares it with its own past. Each task's difficulty comes from how every model did on it. A model that gets worse under the same name shows up here, with a date.

Baselines are still building. No verdicts yet.

Models watched
10
With a verdict
0 of 10
Degraded now
–
Answers graded
145

Task raceOne task, every model, on one clock

  • Read
  • Edit
  • Command
  • Test run
  • Message
  • Finish (pass)
  • Finish (fail or out of time)

Each lane is one model working on the same task, starting at zero and ending where it finished. Every mark is one action it took.

Task 1, first run 4 Oct

Each model got the same issue in the same codebase. A change passes when the task's tests pass and nothing that passed before breaks. Real fix: 133 lines.

10 of 10 passed. Best on this task: GPT-6 Luna. Fastest pass: Claude Sonnet 5.5 in 1m 36s. Smallest passing change: GPT-6.1 Sol, 100 lines against 133 in the real fix.

  1. GPT-6 LunaCodex

    Passed in 2m 23s. 13 reads, 5 edits, 1 command, 5 test runs, 3 messages. Edited again after a test run 3 times. Changed 116 lines in 3 files; the real fix was 133 lines. Task Elo 1778. Elo 1778 after this task, up 278. Used 4.9k output tokens, 1.36M in all. Estimated cost $0.024 at list price.

  2. GPT-6.1 SolCodex

    Passed in 2m 24s. 14 reads, 4 edits, 1 command, 3 test runs, 5 messages. Edited again after a test run once. Changed 100 lines in 3 files; the real fix was 133 lines. Task Elo 1752. Elo 1752 after this task, up 252. Used 6.1k output tokens, 900k in all. Estimated cost $0.280 at list price.

  3. GPT-6 AstraCodex

    Passed in 4m 24s. 21 reads, 3 edits, 2 commands, 4 test runs, 5 messages. Edited again after a test run 2 times. Changed 113 lines in 4 files; the real fix was 133 lines. Task Elo 1596. Elo 1596 after this task, up 96. Used 5.8k output tokens, 1.20M in all. Estimated cost $2.02 at list price.

  4. Claude Sonnet 5.5Claude Code

    Passed in 1m 36s. 16 reads, 3 edits, 1 test run, 3 messages. Edited again after a test run once. Changed 120 lines in 4 files; the real fix was 133 lines. Task Elo 1557. Elo 1557 after this task, up 57. Used 9.7k output tokens, 854k in all. Estimated cost $0.385 at list price.

  5. Claude Haiku 5.5Claude Code

    Passed in 9m 28s. 30 reads, 16 edits, 7 commands, 13 test runs, 18 messages. Edited again after a test run 2 times. Changed 119 lines in 5 files; the real fix was 133 lines. Task Elo 1481. Elo 1481 after this task, down 19. Used 55k output tokens, 5.11M in all. Estimated cost $0.090 at list price.

  6. GPT-6 Luna (high)Codex

    Passed in 3m 9s. 16 reads, 6 edits, 2 commands, 3 test runs, 5 messages. Edited again after a test run 2 times. Changed 141 lines in 3 files; the real fix was 133 lines. Task Elo 1462. Elo 1462 after this task, down 38. Used 11k output tokens, 2.12M in all. Estimated cost $0.036 at list price.

  7. DeepSeek V4.1 FlashDeepSeek Harness

    Passed in 7m 27s. 65 reads, 12 edits, 5 commands, 12 test runs, 1 message. Edited again after a test run once. Changed 132 lines in 5 files; the real fix was 133 lines. Task Elo 1462. Elo 1462 after this task, down 38. Used 57k output tokens, 7.94M in all. Estimated cost $0.073 at list price.

  8. Grok 4.7Cursor

    Passed in 8m 32s. 79 reads, 10 edits, 4 commands, 1 test run, 2 messages. Changed 117 lines in 3 files; the real fix was 133 lines. Task Elo 1443. Elo 1443 after this task, down 57. Used 31k output tokens, 3.75M in all. Estimated cost $2.53 at list price.

  9. Claude Opus 5.5Claude Code

    Passed in 6m 26s. 7 reads, 5 edits, 10 commands, 6 test runs, 5 messages. Edited again after a test run 3 times. Changed 157 lines in 6 files; the real fix was 133 lines. Task Elo 1274. Elo 1274 after this task, down 226. Used 17k output tokens, 1.62M in all. Estimated cost $1.34 at list price.

  10. Kimi K3Kimi Code

    Passed in 11m 9s. 39 reads, 11 edits, 4 commands, 6 test runs, 1 message. Edited again after a test run 2 times. Changed 136 lines in 5 files; the real fix was 133 lines. Task Elo 1192. Elo 1192 after this task, down 308. Used 20k output tokens, 4.04M in all. Estimated cost $12.37 at list price.

Task 2, first run 4 Oct

Each model got the same issue in the same codebase. A change passes when the task's tests pass and nothing that passed before breaks. Real fix: 69 lines.

9 of 10 passed. Best on this task: GPT-6.1 Sol. Fastest pass: Claude Opus 5.5 in 3m 1s. Smallest passing change: GPT-6.1 Sol, 65 lines against 69 in the real fix.

  1. GPT-6.1 SolCodex

    Passed in 3m 10s. 23 reads, 1 edit, 2 commands, 3 test runs, 6 messages. Changed 65 lines in 4 files; the real fix was 69 lines. Task Elo 1771. Elo 1765 after this task, up 13. Used 5.4k output tokens, 1.21M in all. Estimated cost $0.280 at list price.

  2. Claude Haiku 5.5Claude Code

    Passed in 9m 7s. 37 reads, 11 edits, 10 commands, 18 test runs, 22 messages. Edited again after a test run once. Changed 66 lines in 4 files; the real fix was 69 lines. Task Elo 1634. Elo 1556 after this task, up 75. Used 51k output tokens, 4.82M in all. Estimated cost $0.084 at list price.

  3. GPT-6 AstraCodex

    Passed in 4m 55s. 21 reads, 4 edits, 3 commands, 7 test runs, 5 messages. Edited again after a test run 2 times. Changed 69 lines in 4 files; the real fix was 69 lines. Task Elo 1594. Elo 1594 after this task, down 2. Used 5.4k output tokens, 1.25M in all. Estimated cost $2.03 at list price.

  4. DeepSeek V4.1 FlashDeepSeek Harness

    Passed in 5m 45s. 51 reads, 11 edits, 9 commands, 10 test runs, 1 message. Edited again after a test run 2 times. Changed 76 lines in 4 files; the real fix was 69 lines. Task Elo 1575. Elo 1519 after this task, up 57. Used 40k output tokens, 5.35M in all. Estimated cost $0.052 at list price.

  5. Claude Opus 5.5Claude Code

    Passed in 3m 1s. 12 reads, 2 edits, 4 test runs, 3 messages. Edited again after a test run once. Changed 77 lines in 4 files; the real fix was 69 lines. Task Elo 1556. Elo 1425 after this task, up 151. Used 9.3k output tokens, 590k in all. Estimated cost $0.596 at list price.

  6. GPT-6 Luna (high)Codex

    Passed in 4m 1s. 26 reads, 8 edits, 3 commands, 8 test runs, 6 messages. Edited again after a test run 3 times. Changed 91 lines in 4 files; the real fix was 69 lines. Task Elo 1500. Elo 1482 after this task, up 20. Used 14k output tokens, 2.99M in all. Estimated cost $0.046 at list price.

  7. GPT-6 LunaCodex

    Passed in 7m 17s. 28 reads, 10 edits, 3 commands, 7 test runs, 7 messages. Edited again after a test run 2 times. Changed 85 lines in 4 files; the real fix was 69 lines. Task Elo 1462. Elo 1604 after this task, down 174. Used 13k output tokens, 2.89M in all. Estimated cost $0.048 at list price.

  8. Kimi K3Kimi Code

    Passed in 9m 50s. 19 reads, 11 edits, 5 commands, 10 test runs, 1 message. Edited again after a test run once. Changed 73 lines in 4 files; the real fix was 69 lines. Task Elo 1423. Elo 1324 after this task, up 132. Used 15k output tokens, 5.19M in all. Estimated cost $15.76 at list price.

  9. Grok 4.7Cursor

    Passed in 9m 47s. 73 reads, 11 edits, 4 commands, 5 test runs, 3 messages. Edited again after a test run once. Changed 81 lines in 4 files; the real fix was 69 lines. Task Elo 1339. Elo 1397 after this task, down 46. Used 37k output tokens, 4.67M in all. Estimated cost $3.10 at list price.

  10. Claude Sonnet 5.5Claude Code

    Failed in 2m 8s. 12 reads, 13 edits, 1 command, 4 test runs, 4 messages. Edited again after a test run 2 times. Changed 74 lines in 4 files; the real fix was 69 lines. Task Elo 993. Elo 1356 after this task, down 201. Used 12k output tokens, 574k in all. Estimated cost $0.308 at list price.

Task 3, first run 4 Oct

Each model got the same issue in the same codebase. A change passes when the task's tests pass and nothing that passed before breaks. Real fix: 41 lines.

7 of 10 passed. Best on this task: GPT-6.1 Sol. Fastest pass: GPT-6.1 Sol in 1m 41s. Smallest passing change: Kimi K3, 27 lines against 41 in the real fix.

  1. GPT-6.1 SolCodex

    Passed in 1m 41s. 8 reads, 3 edits, 2 commands, 3 test runs, 5 messages. Edited again after a test run once. Changed 29 lines in 2 files; the real fix was 41 lines. Task Elo 1797. Elo 1772 after this task, up 7. Used 3.3k output tokens, 536k in all. Estimated cost $0.152 at list price.

  2. Claude Opus 5.5Claude Code

    Passed in 1m 50s. 7 reads, 1 edit, 3 test runs, 2 messages. Changed 29 lines in 2 files; the real fix was 41 lines. Task Elo 1720. Elo 1512 after this task, up 87. Used 5.6k output tokens, 324k in all. Estimated cost $0.353 at list price.

  3. Kimi K3Kimi Code

    Passed in 6m 55s. 14 reads, 6 edits, 2 commands, 6 test runs, 1 message. Edited again after a test run once. Changed 27 lines in 2 files; the real fix was 41 lines. Task Elo 1629. Elo 1426 after this task, up 102. Used 9.0k output tokens, 1.62M in all. Estimated cost $4.96 at list price.

  4. GPT-6 Luna (high)Codex

    Passed in 1m 46s. 12 reads, 2 edits, 2 commands, 4 test runs, 5 messages. Changed 64 lines in 2 files; the real fix was 41 lines. Task Elo 1607. Elo 1518 after this task, up 36. Used 5.8k output tokens, 924k in all. Estimated cost $0.018 at list price.

  5. GPT-6 AstraCodex

    Passed in 2m 51s. 9 reads, 3 edits, 1 command, 3 test runs, 4 messages. Edited again after a test run once. Changed 32 lines in 2 files; the real fix was 41 lines. Task Elo 1607. Elo 1594 after this task, unchanged. Used 2.7k output tokens, 429k in all. Estimated cost $0.819 at list price.

  6. DeepSeek V4.1 FlashDeepSeek Harness

    Passed in 3m 49s. 33 reads, 6 edits, 4 commands, 7 test runs, 1 message. Edited again after a test run 2 times. Changed 48 lines in 2 files; the real fix was 41 lines. Task Elo 1586. Elo 1537 after this task, up 18. Used 24k output tokens, 2.57M in all. Estimated cost $0.031 at list price.

  7. Grok 4.7Cursor

    Passed in 3m 44s. 35 reads, 9 edits, 1 command, 3 test runs, 3 messages. Edited again after a test run once. Changed 61 lines in 2 files; the real fix was 41 lines. Task Elo 1478. Elo 1426 after this task, up 29. Used 15k output tokens, 1.74M in all. Estimated cost $1.09 at list price.

  8. GPT-6 LunaCodex

    Failed in 1m 39s. 8 reads, 2 edits, 1 command, 1 test run, 3 messages. Changed 20 lines in 2 files; the real fix was 41 lines. Task Elo 1123. Elo 1489 after this task, down 115. Used 2.1k output tokens, 333k in all. Estimated cost $0.0094 at list price.

  9. Claude Sonnet 5.5Claude Code

    Out of time at 15m 0s. 6 reads, 1 edit, 5 commands, 4 test runs, 3 messages. Task Elo 1123. Elo 1294 after this task, down 62.

  10. Claude Haiku 5.5Claude Code

    Out of time at 15m 0s. 24 reads, 10 edits, 4 commands, 9 test runs, 13 messages. Edited again after a test run 3 times. Task Elo 1123. Elo 1455 after this task, down 101.

Task 4, first run 4 Oct

Each model got the same issue in the same codebase. A change passes when the task's tests pass and nothing that passed before breaks. Real fix: 77 lines.

10 of 10 passed. Best on this task: GPT-6.1 Sol and GPT-6 Luna. Fastest pass: GPT-6.1 Sol in 2m 22s. Smallest passing change: GPT-6 Astra, 44 lines against 77 in the real fix.

  1. GPT-6.1 SolCodex

    Passed in 2m 22s. 13 reads, 3 edits, 2 commands, 4 test runs, 6 messages. Edited again after a test run 2 times. Changed 52 lines in 1 file; the real fix was 77 lines. Task Elo 1720. Elo 1764 after this task, down 8. Used 3.6k output tokens, 944k in all. Estimated cost $0.221 at list price.

  2. GPT-6 LunaCodex

    Passed in 3m 48s. 9 reads, 2 edits, 1 command, 5 test runs, 5 messages. Changed 65 lines in 1 file; the real fix was 77 lines. Task Elo 1720. Elo 1545 after this task, up 56. Used 4.8k output tokens, 827k in all. Estimated cost $0.019 at list price.

  3. GPT-6 AstraCodex

    Passed in 4m 11s. 18 reads, 1 edit, 3 commands, 4 test runs, 5 messages. Changed 44 lines in 1 file; the real fix was 77 lines. Task Elo 1613. Elo 1601 after this task, up 7. Used 3.5k output tokens, 1.07M in all. Estimated cost $1.77 at list price.

  4. DeepSeek V4.1 FlashDeepSeek Harness

    Passed in 5m 23s. 32 reads, 4 edits, 5 commands, 5 test runs, 1 message. Edited again after a test run once. Changed 65 lines in 1 file; the real fix was 77 lines. Task Elo 1575. Elo 1547 after this task, up 10. Used 25k output tokens, 2.44M in all. Estimated cost $0.033 at list price.

  5. Claude Haiku 5.5Claude Code

    Passed in 3m 49s. 7 reads, 5 edits, 2 commands, 7 test runs, 6 messages. Edited again after a test run once. Changed 70 lines in 1 file; the real fix was 77 lines. Task Elo 1500. Elo 1464 after this task, up 9. Used 20k output tokens, 929k in all. Estimated cost $0.026 at list price.

  6. GPT-6 Luna (high)Codex

    Passed in 3m 29s. 9 reads, 2 edits, 3 commands, 8 test runs, 6 messages. Edited again after a test run once. Changed 74 lines in 1 file; the real fix was 77 lines. Task Elo 1463. Elo 1505 after this task, down 13. Used 9.0k output tokens, 2.16M in all. Estimated cost $0.034 at list price.

  7. Claude Opus 5.5Claude Code

    Passed in 4m 24s. 7 reads, 5 edits, 2 commands, 6 test runs, 3 messages. Edited again after a test run once. Changed 67 lines in 1 file; the real fix was 77 lines. Task Elo 1463. Elo 1500 after this task, down 12. Used 8.6k output tokens, 490k in all. Estimated cost $0.525 at list price.

  8. Claude Sonnet 5.5Claude Code

    Passed in 4m 42s. 9 reads, 1 edit, 4 test runs, 4 messages. Changed 68 lines in 1 file; the real fix was 77 lines. Task Elo 1425. Elo 1328 after this task, up 34. Used 5.7k output tokens, 354k in all. Estimated cost $0.191 at list price.

  9. Kimi K3Kimi Code

    Passed in 13m 20s. 12 reads, 5 edits, 2 commands, 10 test runs, 1 message. Changed 67 lines in 1 file; the real fix was 77 lines. Task Elo 1322. Elo 1400 after this task, down 26. Used 14k output tokens, 2.02M in all. Estimated cost $6.22 at list price.

  10. Grok 4.7Cursor

    Passed in 6m 21s. 33 reads, 4 edits, 1 command, 2 test runs, 4 messages. Changed 79 lines in 1 file; the real fix was 77 lines. Task Elo 1162. Elo 1370 after this task, down 56. Used 19k output tokens, 1.37M in all. Estimated cost $1.32 at list price.

Task 5, first run 4 Oct

Each model got the same issue in the same codebase. A change passes when the task's tests pass and nothing that passed before breaks. Real fix: 156 lines.

8 of 9 passed. Best on this task: GPT-6.1 Sol. Fastest pass: GPT-6.1 Sol in 2m 30s. Smallest passing change: GPT-6.1 Sol and GPT-6 Astra, 98 lines against 156 in the real fix.

  1. GPT-6.1 SolCodex

    Passed in 2m 30s. 18 reads, 4 edits, 2 commands, 3 test runs, 6 messages. Changed 98 lines in 2 files; the real fix was 156 lines. Task Elo 1744. Elo 1768 after this task, up 4. Used 4.9k output tokens, 1.28M in all. Estimated cost $0.300 at list price.

  2. GPT-6 AstraCodex

    Passed in 3m 44s. 18 reads, 3 edits, 3 commands, 4 test runs, 5 messages. Edited again after a test run once. Changed 98 lines in 2 files; the real fix was 156 lines. Task Elo 1646. Elo 1613 after this task, up 12. Used 4.5k output tokens, 1.11M in all. Estimated cost $1.89 at list price.

  3. GPT-6 Luna (high)Codex

    Passed in 5m 25s. 30 reads, 11 edits, 4 commands, 9 test runs, 6 messages. Edited again after a test run 4 times. Changed 188 lines in 2 files; the real fix was 156 lines. Task Elo 1541. Elo 1514 after this task, up 9. Used 19k output tokens, 4.55M in all. Estimated cost $0.068 at list price.

  4. GPT-6 LunaCodex

    Passed in 12m 31s. 37 reads, 14 edits, 7 commands, 11 test runs, 7 messages. Edited again after a test run 6 times. Changed 138 lines in 2 files; the real fix was 156 lines. Task Elo 1541. Elo 1546 after this task, up 1. Used 15k output tokens, 5.34M in all. Estimated cost $0.075 at list price.

  5. Claude Haiku 5.5Claude Code

    Passed in 7m 28s. 43 reads, 11 edits, 2 commands, 9 test runs, 38 messages. Edited again after a test run 3 times. Changed 145 lines in 3 files; the real fix was 156 lines. Task Elo 1520. Elo 1476 after this task, up 12. Used 79k output tokens, 6.35M in all. Estimated cost $0.117 at list price.

  6. Claude Sonnet 5.5Claude Code

    Passed in 7m 53s. 22 reads, 3 edits, 9 commands, 9 test runs, 6 messages. Edited again after a test run 2 times. Changed 126 lines in 3 files; the real fix was 156 lines. Task Elo 1520. Elo 1369 after this task, up 41. Used 27k output tokens, 3.21M in all. Estimated cost $1.12 at list price.

  7. Claude Opus 5.5Claude Code

    Passed in 7m 2s. 15 reads, 2 edits, 5 test runs, 3 messages. Edited again after a test run once. Changed 170 lines in 2 files; the real fix was 156 lines. Task Elo 1459. Elo 1494 after this task, down 6. Used 18k output tokens, 880k in all. Estimated cost $1.02 at list price.

  8. DeepSeek V4.1 FlashDeepSeek Harness

    Passed in 8m 46s. 84 reads, 9 edits, 6 commands, 9 test runs, 1 message. Edited again after a test run 3 times. Changed 234 lines in 3 files; the real fix was 156 lines. Task Elo 1371. Elo 1518 after this task, down 29. Used 68k output tokens, 8.62M in all. Estimated cost $0.082 at list price.

  9. Grok 4.7Cursor

    Out of time at 15m 0s. 63 reads, 6 edits, 1 test run, 2 messages. Task Elo 1022. Elo 1328 after this task, down 42.

Task 6, first run 4 Oct

Each model got the same issue in the same codebase. A change passes when the task's tests pass and nothing that passed before breaks. Real fix: 45 lines.

9 of 9 passed. Best on this task: GPT-6 Luna (high). Fastest pass: GPT-6.1 Sol in 2m 15s. Smallest passing change: GPT-6 Luna (high) and GPT-6 Luna, 27 lines against 45 in the real fix.

  1. GPT-6 Luna (high)Codex

    Passed in 3m 43s. 17 reads, 2 edits, 1 command, 7 test runs, 6 messages. Changed 27 lines in 1 file; the real fix was 45 lines. Task Elo 1794. Elo 1553 after this task, up 39. Used 7.9k output tokens, 1.08M in all. Estimated cost $0.020 at list price.

  2. GPT-6 LunaCodex

    Passed in 4m 12s. 16 reads, 3 edits, 7 commands, 5 test runs, 4 messages. Edited again after a test run 2 times. Changed 27 lines in 1 file; the real fix was 45 lines. Task Elo 1708. Elo 1571 after this task, up 25. Used 5.1k output tokens, 1.39M in all. Estimated cost $0.023 at list price.

  3. GPT-6.1 SolCodex

    Passed in 2m 15s. 17 reads, 1 edit, 2 commands, 5 test runs, 5 messages. Edited again after a test run once. Changed 31 lines in 1 file; the real fix was 45 lines. Task Elo 1683. Elo 1756 after this task, down 12. Used 3.9k output tokens, 905k in all. Estimated cost $0.226 at list price.

  4. GPT-6 AstraCodex

    Passed in 3m 12s. 18 reads, 1 edit, 1 command, 3 test runs, 4 messages. Changed 32 lines in 1 file; the real fix was 45 lines. Task Elo 1522. Elo 1600 after this task, down 13. Used 3.3k output tokens, 885k in all. Estimated cost $1.45 at list price.

  5. Claude Sonnet 5.5Claude Code

    Passed in 2m 29s. 16 reads, 1 edit, 7 test runs, 8 messages. Edited again after a test run once. Changed 36 lines in 2 files; the real fix was 45 lines. Task Elo 1500. Elo 1392 after this task, up 23. Used 8.4k output tokens, 821k in all. Estimated cost $0.327 at list price.

  6. Claude Haiku 5.5Claude Code

    Passed in 3m 57s. 14 reads, 6 edits, 1 command, 11 test runs, 13 messages. Edited again after a test run 2 times. Changed 37 lines in 1 file; the real fix was 45 lines. Task Elo 1456. Elo 1474 after this task, down 2. Used 20k output tokens, 1.09M in all. Estimated cost $0.026 at list price.

  7. Claude Opus 5.5Claude Code

    Passed in 5m 43s. 7 reads, 1 edit, 6 test runs, 3 messages. Edited again after a test run once. Changed 33 lines in 1 file; the real fix was 45 lines. Task Elo 1387. Elo 1481 after this task, down 13. Used 7.0k output tokens, 423k in all. Estimated cost $0.461 at list price.

  8. DeepSeek V4.1 FlashDeepSeek Harness

    Passed in 5m 37s. 50 reads, 5 edits, 5 commands, 6 test runs, 1 message. Edited again after a test run once. Changed 39 lines in 1 file; the real fix was 45 lines. Task Elo 1337. Elo 1494 after this task, down 24. Used 39k output tokens, 4.84M in all. Estimated cost $0.051 at list price.

  9. Grok 4.7Cursor

    Passed in 6m 41s. 62 reads, 6 edits, 4 test runs, 5 messages. Edited again after a test run once. Changed 62 lines in 1 file; the real fix was 45 lines. Task Elo 1004. Elo 1294 after this task, down 34. Used 23k output tokens, 3.18M in all. Estimated cost $1.89 at list price.

Task 1, first run 5 Oct

Each model got the same issue in the same codebase. A change passes when the task's tests pass and nothing that passed before breaks. Real fix: 90 lines.

8 of 9 passed. Best on this task: GPT-6 Luna (high). Fastest pass: GPT-6.1 Sol in 1m 33s. Smallest passing change: GPT-6 Luna (high) and GPT-6 Astra, 34 lines against 90 in the real fix.

  1. GPT-6 Luna (high)Codex

    Passed in 2m 2s. 14 reads, 3 edits, 2 commands, 4 test runs, 6 messages. Changed 34 lines in 3 files; the real fix was 90 lines. Task Elo 1884. Elo 1588 after this task, up 35. Used 8.1k output tokens, 1.00M in all. Estimated cost $0.019 at list price.

  2. GPT-6.1 SolCodex

    Passed in 1m 33s. 14 reads, 1 edit, 2 commands, 3 test runs, 4 messages. Changed 41 lines in 3 files; the real fix was 90 lines. Task Elo 1720. Elo 1750 after this task, down 6. Used 3.1k output tokens, 572k in all. Estimated cost $0.171 at list price.

  3. GPT-6 AstraCodex

    Passed in 2m 10s. 9 reads, 3 edits, 2 commands, 5 test runs, 5 messages. Edited again after a test run once. Changed 34 lines in 3 files; the real fix was 90 lines. Task Elo 1642. Elo 1606 after this task, up 6. Used 3.5k output tokens, 652k in all. Estimated cost $1.60 at list price.

  4. DeepSeek V4.1 FlashDeepSeek Harness

    Passed in 3m 38s. 49 reads, 5 edits, 2 commands, 7 test runs, 1 message. Edited again after a test run 2 times. Changed 54 lines in 3 files; the real fix was 90 lines. Task Elo 1618. Elo 1510 after this task, up 16. Used 37k output tokens, 3.86M in all. Estimated cost $0.043 at list price.

  5. Claude Haiku 5.5Claude Code

    Passed in 4m 1s. 16 reads, 4 edits, 1 command, 13 test runs, 24 messages. Edited again after a test run 2 times. Changed 84 lines in 3 files; the real fix was 90 lines. Task Elo 1453. Elo 1474 after this task, unchanged. Used 29k output tokens, 1.46M in all. Estimated cost $0.035 at list price.

  6. Claude Sonnet 5.5Claude Code

    Passed in 3m 56s. 10 reads, 7 edits, 2 commands, 4 test runs, 4 messages. Changed 73 lines in 3 files; the real fix was 90 lines. Task Elo 1429. Elo 1401 after this task, up 9. Used 9.4k output tokens, 686k in all. Estimated cost $0.316 at list price.

  7. Grok 4.7Cursor

    Passed in 9m 39s. 55 reads, 5 edits, 1 command, 2 test runs, 6 messages. Edited again after a test run once. Changed 64 lines in 3 files; the real fix was 90 lines. Task Elo 1429. Elo 1318 after this task, up 24. Used 1.4k output tokens, 416k in all. Estimated cost $0.219 at list price.

  8. Claude Opus 5.5Claude Code

    Passed in 4m 21s. 10 reads, 6 edits, 7 test runs, 7 messages. Edited again after a test run once. Changed 86 lines in 3 files; the real fix was 90 lines. Task Elo 1239. Elo 1457 after this task, down 24. Used 9.0k output tokens, 500k in all. Estimated cost $0.508 at list price.

  9. GPT-6 LunaCodex

    Failed in 25s. 7 reads, 1 edit, 1 test run, 3 messages. Changed 3 lines in 1 file; the real fix was 90 lines. Task Elo 980. Elo 1516 after this task, down 55. Used 1.2k output tokens, 231k in all. Estimated cost $0.0052 at list price.

Task 2, first run 5 Oct

Each model got the same issue in the same codebase. A change passes when the task's tests pass and nothing that passed before breaks. Real fix: 16 lines.

7 of 9 passed. Best on this task: Claude Haiku 5.5. Fastest pass: GPT-6 Astra in 1m 23s. Smallest passing change: Claude Haiku 5.5, GPT-6 Astra, GPT-6.1 Sol, Claude Opus 5.5 and DeepSeek V4.1 Flash, 16 lines against 16 in the real fix.

  1. Claude Haiku 5.5Claude Code

    Passed in 1m 24s. 12 reads, 7 edits, 3 commands, 6 test runs, 14 messages. Edited again after a test run once. Changed 16 lines in 4 files; the real fix was 16 lines. Task Elo 1735. Elo 1505 after this task, up 31. Used 12k output tokens, 772k in all. Estimated cost $0.018 at list price.

  2. GPT-6 AstraCodex

    Passed in 1m 23s. 8 reads, 2 edits, 2 test runs, 4 messages. Edited again after a test run once. Changed 16 lines in 4 files; the real fix was 16 lines. Task Elo 1660. Elo 1613 after this task, up 7. Used 1.7k output tokens, 359k in all. Estimated cost $0.695 at list price.

  3. GPT-6.1 SolCodex

    Passed in 1m 34s. 9 reads, 3 edits, 3 test runs, 5 messages. Edited again after a test run once. Changed 16 lines in 4 files; the real fix was 16 lines. Task Elo 1660. Elo 1739 after this task, down 11. Used 3.0k output tokens, 441k in all. Estimated cost $0.129 at list price.

  4. Claude Opus 5.5Claude Code

    Passed in 1m 55s. 8 reads, 9 edits, 6 test runs, 6 messages. Edited again after a test run once. Changed 16 lines in 4 files; the real fix was 16 lines. Task Elo 1613. Elo 1476 after this task, up 19. Used 5.1k output tokens, 238k in all. Estimated cost $0.265 at list price.

  5. DeepSeek V4.1 FlashDeepSeek Harness

    Passed in 3m 10s. 24 reads, 8 edits, 1 command, 14 test runs, 1 message. Edited again after a test run once. Changed 16 lines in 4 files; the real fix was 16 lines. Task Elo 1591. Elo 1520 after this task, up 10. Used 15k output tokens, 1.69M in all. Estimated cost $0.023 at list price.

  6. GPT-6 Luna (high)Codex

    Passed in 2m 3s. 9 reads, 2 edits, 5 test runs, 5 messages. Edited again after a test run once. Changed 18 lines in 5 files; the real fix was 16 lines. Task Elo 1523. Elo 1581 after this task, down 7. Used 6.4k output tokens, 661k in all. Estimated cost $0.016 at list price.

  7. Grok 4.7Cursor

    Passed in 3m 3s. 48 reads, 8 edits, 1 test run, 3 messages. Changed 18 lines in 5 files; the real fix was 16 lines. Task Elo 1351. Elo 1326 after this task, up 8. Used 9.5k output tokens, 1.51M in all. Estimated cost $1.11 at list price.

  8. Claude Sonnet 5.5Claude Code

    Failed in 1m 14s. 4 reads, 10 edits, 6 test runs, 4 messages. Edited again after a test run once. Changed 16 lines in 4 files; the real fix was 16 lines. 1 test that passed before broke. Task Elo 1075. Elo 1375 after this task, down 26. Used 4.8k output tokens, 266k in all. Estimated cost $0.142 at list price.

  9. GPT-6 LunaCodex

    Failed in 1m 42s. 7 reads, 1 edit, 2 commands, 4 test runs, 4 messages. Changed 21 lines in 4 files; the real fix was 16 lines. Task Elo 1075. Elo 1481 after this task, down 35. Used 4.5k output tokens, 955k in all. Estimated cost $0.018 at list price.

Task 3, first run 5 Oct

Each model got the same issue in the same codebase. A change passes when the task's tests pass and nothing that passed before breaks. Real fix: 131 lines.

9 of 9 passed. Best on this task: GPT-6 Luna. Fastest pass: Claude Sonnet 5.5 in 1m 47s. Smallest passing change: GPT-6 Luna, 83 lines against 131 in the real fix.

  1. GPT-6 LunaCodex

    Passed in 3m 18s. 14 reads, 6 edits, 2 commands, 3 test runs, 4 messages. Edited again after a test run 2 times. Changed 83 lines in 3 files; the real fix was 131 lines. Task Elo 1823. Elo 1514 after this task, up 33. Used 4.9k output tokens, 852k in all. Estimated cost $0.018 at list price.

  2. GPT-6 Luna (high)Codex

    Passed in 1m 56s. 14 reads, 5 edits, 2 commands, 5 test runs, 4 messages. Edited again after a test run once. Changed 142 lines in 3 files; the real fix was 131 lines. Task Elo 1680. Elo 1592 after this task, up 11. Used 7.9k output tokens, 975k in all. Estimated cost $0.019 at list price.

  3. GPT-6.1 SolCodex

    Passed in 2m 29s. 11 reads, 4 edits, 2 commands, 4 test runs, 6 messages. Edited again after a test run once. Changed 101 lines in 3 files; the real fix was 131 lines. Task Elo 1632. Elo 1727 after this task, down 12. Used 5.7k output tokens, 790k in all. Estimated cost $0.236 at list price.

  4. GPT-6 AstraCodex

    Passed in 5m 5s. 10 reads, 3 edits, 4 commands, 6 test runs, 5 messages. Changed 89 lines in 3 files; the real fix was 131 lines. Task Elo 1543. Elo 1606 after this task, down 7. Used 4.9k output tokens, 655k in all. Estimated cost $1.32 at list price.

  5. Claude Sonnet 5.5Claude Code

    Passed in 1m 47s. 11 reads, 6 edits, 2 test runs, 4 messages. Changed 150 lines in 3 files; the real fix was 131 lines. Task Elo 1521. Elo 1392 after this task, up 17. Used 12k output tokens, 609k in all. Estimated cost $0.319 at list price.

  6. Claude Opus 5.5Claude Code

    Passed in 3m 37s. 13 reads, 3 edits, 1 command, 3 test runs, 4 messages. Changed 147 lines in 3 files; the real fix was 131 lines. Task Elo 1457. Elo 1475 after this task, down 1. Used 10k output tokens, 873k in all. Estimated cost $0.737 at list price.

  7. Claude Haiku 5.5Claude Code

    Passed in 6m 2s. 27 reads, 8 edits, 5 commands, 8 test runs, 25 messages. Edited again after a test run once. Changed 173 lines in 3 files; the real fix was 131 lines. Task Elo 1367. Elo 1492 after this task, down 13. Used 45k output tokens, 2.81M in all. Estimated cost $0.060 at list price.

  8. DeepSeek V4.1 FlashDeepSeek Harness

    Passed in 5m 19s. 51 reads, 7 edits, 5 commands, 10 test runs, 1 message. Edited again after a test run once. Changed 244 lines in 3 files; the real fix was 131 lines. Task Elo 1318. Elo 1502 after this task, down 18. Used 46k output tokens, 5.17M in all. Estimated cost $0.055 at list price.

  9. Grok 4.7Cursor

    Passed in 8m 13s. 69 reads, 8 edits, 1 command, 2 test runs, 4 messages. Changed 200 lines in 3 files; the real fix was 131 lines. Task Elo 1133. Elo 1310 after this task, down 16. Used 31k output tokens, 3.54M in all. Estimated cost $2.69 at list price.

Task 1, first run 8 Oct

Each model got the same issue in the same codebase. A change passes when the task's tests pass and nothing that passed before breaks. Real fix: 36 lines.

7 of 7 passed. Best on this task: GPT-6 Luna. Fastest pass: GPT-6 Luna in 40s. Smallest passing change: GPT-6.1 Sol and GPT-6 Astra, 14 lines against 36 in the real fix.

  1. GPT-6 LunaCodex

    Passed in 40s. 8 reads, 1 edit, 1 command, 3 test runs, 4 messages. Changed 16 lines in 1 file; the real fix was 36 lines. Task Elo 1716. Elo 1532 after this task, up 18. Used 2.2k output tokens, 369k in all. Estimated cost $0.0083 at list price.

  2. GPT-6.1 SolCodex

    Passed in 2m 4s. 12 reads, 1 edit, 2 commands, 4 test runs, 5 messages. Edited again after a test run once. Changed 14 lines in 1 file; the real fix was 36 lines. Task Elo 1629. Elo 1722 after this task, down 5. Used 2.7k output tokens, 601k in all. Estimated cost $0.173 at list price.

  3. GPT-6 AstraCodex

    Passed in 2m 9s. 12 reads, 2 edits, 1 command, 3 test runs, 5 messages. Edited again after a test run once. Changed 14 lines in 1 file; the real fix was 36 lines. Task Elo 1550. Elo 1604 after this task, down 2. Used 2.5k output tokens, 669k in all. Estimated cost $1.14 at list price.

  4. Claude Sonnet 5.5Claude Code

    Passed in 56s. 12 reads, 1 edit, 2 test runs, 2 messages. Changed 19 lines in 1 file; the real fix was 36 lines. Task Elo 1525. Elo 1407 after this task, up 15. Used 4.8k output tokens, 349k in all. Estimated cost $0.171 at list price.

  5. Claude Haiku 5.5Claude Code

    Passed in 3m 18s. 20 reads, 3 edits, 2 commands, 7 test runs, 16 messages. Edited again after a test run once. Changed 19 lines in 1 file; the real fix was 36 lines. Task Elo 1500. Elo 1495 after this task, up 3. Used 15k output tokens, 703k in all. Estimated cost $0.018 at list price.

  6. Claude Opus 5.5Claude Code

    Passed in 3m 21s. 7 reads, 1 edit, 6 test runs, 5 messages. Edited again after a test run once. Changed 22 lines in 1 file; the real fix was 36 lines. Task Elo 1279. Elo 1464 after this task, down 11. Used 5.1k output tokens, 349k in all. Estimated cost $0.334 at list price.

  7. GPT-6 Luna (high)Codex

    Passed in 3m 25s. 16 reads, 2 edits, 7 commands, 6 test runs, 6 messages. Edited again after a test run once. Changed 31 lines in 1 file; the real fix was 36 lines. Task Elo 1279. Elo 1570 after this task, down 22. Used 8.6k output tokens, 2.01M in all. Estimated cost $0.032 at list price.

Task 2, first run 8 Oct

Each model got the same issue in the same codebase. A change passes when the task's tests pass and nothing that passed before breaks. Real fix: 52 lines.

7 of 7 passed. Best on this task: GPT-6 Luna. Fastest pass: GPT-6 Luna in 53s. Smallest passing change: GPT-6 Astra, 31 lines against 52 in the real fix.

  1. GPT-6 LunaCodex

    Passed in 53s. 4 reads, 1 edit, 3 commands, 2 test runs, 4 messages. Changed 37 lines in 1 file; the real fix was 52 lines. Task Elo 1746. Elo 1550 after this task, up 18. Used 2.5k output tokens, 370k in all. Estimated cost $0.0077 at list price.

  2. Claude Opus 5.5Claude Code

    Passed in 1m 46s. 6 reads, 1 edit, 1 command, 4 test runs, 3 messages. Changed 37 lines in 1 file; the real fix was 52 lines. Task Elo 1549. Elo 1473 after this task, up 9. Used 5.0k output tokens, 285k in all. Estimated cost $0.319 at list price.

  3. GPT-6 Luna (high)Codex

    Passed in 1m 54s. 7 reads, 1 edit, 5 commands, 5 test runs, 5 messages. Changed 38 lines in 1 file; the real fix was 52 lines. Task Elo 1549. Elo 1571 after this task, up 1. Used 6.0k output tokens, 489k in all. Estimated cost $0.012 at list price.

  4. GPT-6 AstraCodex

    Passed in 2m 0s. 8 reads, 3 edits, 2 commands, 5 test runs, 5 messages. Edited again after a test run once. Changed 31 lines in 1 file; the real fix was 52 lines. Task Elo 1549. Elo 1603 after this task, down 1. Used 2.9k output tokens, 487k in all. Estimated cost $0.932 at list price.

  5. Claude Sonnet 5.5Claude Code

    Passed in 1m 42s. 6 reads, 1 edit, 4 test runs, 3 messages. Changed 48 lines in 2 files; the real fix was 52 lines. Task Elo 1451. Elo 1413 after this task, up 6. Used 4.2k output tokens, 271k in all. Estimated cost $0.144 at list price.

  6. GPT-6.1 SolCodex

    Passed in 2m 2s. 7 reads, 3 edits, 1 command, 5 test runs, 5 messages. Edited again after a test run once. Changed 39 lines in 1 file; the real fix was 52 lines. Task Elo 1375. Elo 1692 after this task, down 30. Used 3.1k output tokens, 498k in all. Estimated cost $0.203 at list price.

  7. Claude Haiku 5.5Claude Code

    Passed in 2m 27s. 8 reads, 5 edits, 3 commands, 9 test runs, 18 messages. Changed 56 lines in 2 files; the real fix was 52 lines. Task Elo 1288. Elo 1483 after this task, down 12. Used 19k output tokens, 802k in all. Estimated cost $0.022 at list price.

Task 3, first run 8 Oct

Each model got the same issue in the same codebase. A change passes when the task's tests pass and nothing that passed before breaks. Real fix: 246 lines.

7 of 7 passed. Best on this task: GPT-6 Luna. Fastest pass: GPT-6 Luna in 2m 13s. Smallest passing change: GPT-6 Luna, 145 lines against 246 in the real fix.

  1. GPT-6 LunaCodex

    Passed in 2m 13s. 15 reads, 9 edits, 3 commands, 6 test runs, 7 messages. Edited again after a test run 3 times. Changed 145 lines in 3 files; the real fix was 246 lines. Task Elo 1940. Elo 1572 after this task, up 22. Used 8.2k output tokens, 1.12M in all. Estimated cost $0.022 at list price.

  2. GPT-6 Luna (high)Codex

    Passed in 2m 24s. 10 reads, 8 edits, 2 commands, 8 test runs, 5 messages. Edited again after a test run 2 times. Changed 149 lines in 4 files; the real fix was 246 lines. Task Elo 1677. Elo 1580 after this task, up 9. Used 9.8k output tokens, 1.70M in all. Estimated cost $0.030 at list price.

  3. GPT-6.1 SolCodex

    Passed in 3m 28s. 8 reads, 5 edits, 4 commands, 3 test runs, 6 messages. Edited again after a test run once. Changed 146 lines in 4 files; the real fix was 246 lines. Task Elo 1556. Elo 1685 after this task, down 7. Used 6.4k output tokens, 717k in all. Estimated cost $0.234 at list price.

  4. GPT-6 AstraCodex

    Passed in 2m 45s. 8 reads, 3 edits, 2 commands, 2 test runs, 5 messages. Edited again after a test run once. Changed 151 lines in 4 files; the real fix was 246 lines. Task Elo 1445. Elo 1595 after this task, down 8. Used 4.9k output tokens, 608k in all. Estimated cost $1.55 at list price.

  5. Claude Sonnet 5.5Claude Code

    Passed in 3m 1s. 6 reads, 3 edits, 5 test runs, 4 messages. Edited again after a test run once. Changed 236 lines in 5 files; the real fix was 246 lines. Task Elo 1418. Elo 1416 after this task, up 3. Used 12k output tokens, 552k in all. Estimated cost $0.314 at list price.

  6. Claude Haiku 5.5Claude Code

    Passed in 3m 16s. 11 reads, 13 edits, 2 commands, 9 test runs, 19 messages. Edited again after a test run once. Changed 241 lines in 5 files; the real fix was 246 lines. Task Elo 1332. Elo 1475 after this task, down 8. Used 39k output tokens, 1.65M in all. Estimated cost $0.044 at list price.

  7. Claude Opus 5.5Claude Code

    Passed in 3m 28s. 2 reads, 3 edits, 6 test runs, 3 messages. Edited again after a test run 2 times. Changed 237 lines in 5 files; the real fix was 246 lines. Task Elo 1234. Elo 1462 after this task, down 11. Used 12k output tokens, 434k in all. Estimated cost $0.552 at list price.

Task 1, first run 9 Oct

Each model got the same issue in the same codebase. A change passes when the task's tests pass and nothing that passed before breaks. Real fix: 11 lines.

4 of 4 passed. Best on this task: GPT-6 Luna and Claude Haiku 5.5. Fastest pass: GPT-6 Luna in 1m 15s. Smallest passing change: Claude Haiku 5.5 and GPT-6.1 Sol, 5 lines against 11 in the real fix.

  1. GPT-6 LunaCodex

    Passed in 1m 15s. 11 reads, 4 edits, 2 commands, 4 test runs, 4 messages. Edited again after a test run once. Changed 11 lines in 2 files; the real fix was 11 lines. Task Elo 1535. Elo 1572 after this task, unchanged. Used 4.1k output tokens, 548k in all. Estimated cost $0.012 at list price.

  2. Claude Haiku 5.5Claude Code

    Passed in 2m 15s. 16 reads, 4 edits, 2 commands, 7 test runs, 16 messages. Edited again after a test run 2 times. Changed 5 lines in 2 files; the real fix was 11 lines. Task Elo 1535. Elo 1479 after this task, up 4. Used 19k output tokens, 1.21M in all. Estimated cost $0.027 at list price.

  3. GPT-6.1 SolCodex

    Passed in 1m 53s. 7 reads, 3 edits, 4 commands, 3 test runs, 5 messages. Edited again after a test run 2 times. Changed 5 lines in 2 files; the real fix was 11 lines. Task Elo 1500. Elo 1678 after this task, down 7. Used 2.8k output tokens, 560k in all. Estimated cost $0.160 at list price.

  4. Claude Sonnet 5.5Claude Code

    Passed in 1m 30s. 5 reads, 1 edit, 6 test runs, 3 messages. Edited again after a test run once. Changed 11 lines in 2 files; the real fix was 11 lines. Task Elo 1429. Elo 1417 after this task, up 1. Used 4.8k output tokens, 253k in all. Estimated cost $0.153 at list price.

Task 2, first run 9 Oct

Each model got the same issue in the same codebase. A change passes when the task's tests pass and nothing that passed before breaks. Real fix: 146 lines.

4 of 4 passed. Best on this task: GPT-6 Luna. Fastest pass: GPT-6.1 Sol in 3m 13s. Smallest passing change: GPT-6 Luna, 60 lines against 146 in the real fix.

  1. GPT-6 LunaCodex

    Passed in 3m 50s. 11 reads, 5 edits, 2 commands, 7 test runs, 6 messages. Edited again after a test run 2 times. Changed 60 lines in 3 files; the real fix was 146 lines. Task Elo 1721. Elo 1579 after this task, up 7. Used 6.0k output tokens, 1.59M in all. Estimated cost $0.025 at list price.

  2. GPT-6.1 SolCodex

    Passed in 3m 13s. 13 reads, 4 edits, 2 commands, 3 test runs, 6 messages. Edited again after a test run 2 times. Changed 92 lines in 3 files; the real fix was 146 lines. Task Elo 1583. Elo 1677 after this task, down 1. Used 5.2k output tokens, 635k in all. Estimated cost $0.197 at list price.

  3. Claude Haiku 5.5Claude Code

    Passed in 5m 19s. 34 reads, 8 edits, 3 commands, 8 test runs, 24 messages. Edited again after a test run once. Changed 98 lines in 3 files; the real fix was 146 lines. Task Elo 1417. Elo 1478 after this task, down 1. Used 39k output tokens, 2.19M in all. Estimated cost $0.048 at list price.

  4. Claude Sonnet 5.5Claude Code

    Passed in 4m 21s. 15 reads, 8 edits, 2 commands, 6 test runs, 6 messages. Edited again after a test run once. Changed 105 lines in 3 files; the real fix was 146 lines. Task Elo 1279. Elo 1412 after this task, down 5. Used 13k output tokens, 822k in all. Estimated cost $0.370 at list price.

Task 3, first run 9 Oct

Each model got the same issue in the same codebase. A change passes when the task's tests pass and nothing that passed before breaks. Real fix: 11 lines.

4 of 4 passed. Best on this task: GPT-6 Luna. Fastest pass: GPT-6 Luna in 23s. Smallest passing change: GPT-6 Luna, Claude Sonnet 5.5 and GPT-6.1 Sol, 6 lines against 11 in the real fix.

  1. GPT-6 LunaCodex

    Passed in 23s. 6 reads, 1 edit, 1 test run, 3 messages. Changed 6 lines in 1 file; the real fix was 11 lines. Task Elo 1658. Elo 1584 after this task, up 5. Used 1.0k output tokens, 207k in all. Estimated cost $0.0046 at list price.

  2. Claude Sonnet 5.5Claude Code

    Passed in 24s. 4 reads, 1 edit, 1 test run, 2 messages. Changed 6 lines in 1 file; the real fix was 11 lines. Task Elo 1537. Elo 1418 after this task, up 6. Used 1.9k output tokens, 100k in all. Estimated cost $0.064 at list price.

  3. GPT-6.1 SolCodex

    Passed in 1m 23s. 7 reads, 3 edits, 2 commands, 3 test runs, 4 messages. Edited again after a test run once. Changed 6 lines in 1 file; the real fix was 11 lines. Task Elo 1425. Elo 1667 after this task, down 10. Used 1.8k output tokens, 327k in all. Estimated cost $0.105 at list price.

  4. Claude Haiku 5.5Claude Code

    Passed in 53s. 6 reads, 3 edits, 1 command, 3 test runs, 10 messages. Edited again after a test run once. Changed 8 lines in 1 file; the real fix was 11 lines. Task Elo 1385. Elo 1475 after this task, down 3. Used 7.0k output tokens, 274k in all. Estimated cost $0.0084 at list price.

Task 1, first run 15 Oct

Each model got the same issue in the same codebase. A change passes when the task's tests pass and nothing that passed before breaks. Real fix: 17 lines.

4 of 4 passed. Best on this task: GPT-6.1 Sol. Fastest pass: Claude Sonnet 5.5 in 33s. Smallest passing change: GPT-6.1 Sol, GPT-6 Astra and Claude Opus 5.5, 13 lines against 17 in the real fix.

  1. GPT-6.1 SolCodex

    Passed in 2m 13s. 11 reads, 1 edit, 1 command, 3 test runs, 4 messages. Changed 13 lines in 1 file; the real fix was 17 lines. Task Elo 1571. Elo 1665 after this task, down 2. Used 2.1k output tokens, 482k in all. Estimated cost $0.128 at list price.

  2. Claude Sonnet 5.5Claude Code

    Passed in 33s. 6 reads, 1 edit, 1 test run, 2 messages. Changed 18 lines in 1 file; the real fix was 17 lines. Task Elo 1500. Elo 1423 after this task, up 5. Used 2.9k output tokens, 150k in all. Estimated cost $0.105 at list price.

  3. GPT-6 AstraCodex

    Passed in 2m 15s. 10 reads, 1 edit, 1 command, 3 test runs, 5 messages. Changed 13 lines in 1 file; the real fix was 17 lines. Task Elo 1465. Elo 1590 after this task, down 4. Used 2.2k output tokens, 450k in all. Estimated cost $0.942 at list price.

  4. Claude Opus 5.5Claude Code

    Passed in 2m 59s. 9 reads, 2 edits, 5 test runs, 4 messages. Edited again after a test run once. Changed 13 lines in 1 file; the real fix was 17 lines. Task Elo 1465. Elo 1463 after this task, up 1. Used 3.5k output tokens, 280k in all. Estimated cost $0.275 at list price.

Task 2, first run 15 Oct

Each model got the same issue in the same codebase. A change passes when the task's tests pass and nothing that passed before breaks. Real fix: 64 lines.

4 of 4 passed. Best on this task: GPT-6.1 Sol. Fastest pass: Claude Sonnet 5.5 in 1m 28s. Smallest passing change: GPT-6.1 Sol and GPT-6 Astra, 37 lines against 64 in the real fix.

  1. GPT-6.1 SolCodex

    Passed in 1m 55s. 9 reads, 1 edit, 3 commands, 3 test runs, 4 messages. Changed 37 lines in 2 files; the real fix was 64 lines. Task Elo 1612. Elo 1666 after this task, up 1. Used 2.8k output tokens, 654k in all. Estimated cost $0.199 at list price.

  2. Claude Sonnet 5.5Claude Code

    Passed in 1m 28s. 6 reads, 4 edits, 5 test runs, 3 messages. Changed 63 lines in 2 files; the real fix was 64 lines. Task Elo 1500. Elo 1426 after this task, up 3. Used 7.3k output tokens, 277k in all. Estimated cost $0.170 at list price.

  3. GPT-6 AstraCodex

    Passed in 2m 8s. 10 reads, 1 edit, 1 command, 3 test runs, 4 messages. Changed 37 lines in 2 files; the real fix was 64 lines. Task Elo 1500. Elo 1588 after this task, down 2. Used 2.4k output tokens, 421k in all. Estimated cost $0.849 at list price.

  4. Claude Opus 5.5Claude Code

    Passed in 3m 10s. 10 reads, 3 edits, 4 test runs, 5 messages. Changed 51 lines in 2 files; the real fix was 64 lines. Task Elo 1388. Elo 1460 after this task, down 3. Used 5.4k output tokens, 361k in all. Estimated cost $0.343 at list price.

Task 3, first run 15 Oct

Each model got the same issue in the same codebase. A change passes when the task's tests pass and nothing that passed before breaks. Real fix: 72 lines.

4 of 4 passed. Best on this task: GPT-6.1 Sol. Fastest pass: Claude Sonnet 5.5 in 1m 30s. Smallest passing change: GPT-6 Astra, 37 lines against 72 in the real fix.

  1. GPT-6.1 SolCodex

    Passed in 2m 14s. 12 reads, 1 edit, 3 commands, 3 test runs, 4 messages. Changed 40 lines in 1 file; the real fix was 72 lines. Task Elo 1612. Elo 1666 after this task, unchanged. Used 3.6k output tokens, 748k in all. Estimated cost $0.206 at list price.

  2. GPT-6 AstraCodex

    Passed in 3m 0s. 14 reads, 1 edit, 1 command, 2 test runs, 6 messages. Changed 37 lines in 1 file; the real fix was 72 lines. Task Elo 1536. Elo 1588 after this task, unchanged. Used 3.0k output tokens, 657k in all. Estimated cost $1.16 at list price.

  3. Claude Sonnet 5.5Claude Code

    Passed in 1m 30s. 6 reads, 2 edits, 1 command, 3 test runs, 4 messages. Edited again after a test run once. Changed 52 lines in 1 file; the real fix was 72 lines. Task Elo 1464. Elo 1428 after this task, up 2. Used 11k output tokens, 322k in all. Estimated cost $0.237 at list price.

  4. Claude Opus 5.5Claude Code

    Passed in 4m 6s. 7 reads, 1 edit, 7 test runs, 5 messages. Edited again after a test run once. Changed 44 lines in 1 file; the real fix was 72 lines. Task Elo 1388. Elo 1458 after this task, down 2. Used 7.3k output tokens, 570k in all. Estimated cost $0.539 at list price.

Task 1, first run 17 Oct

Each model got the same issue in the same codebase. A change passes when the task's tests pass and nothing that passed before breaks. Real fix: 33 lines.

3 of 3 passed. Best on this task: Claude Sonnet 5.5. Fastest pass: Claude Sonnet 5.5 in 1m 2s. Smallest passing change: Claude Sonnet 5.5, 22 lines against 33 in the real fix.

  1. Claude Sonnet 5.5Claude Code

    Passed in 1m 2s. 2 reads, 1 edit, 6 test runs, 2 messages. Changed 22 lines in 1 file; the real fix was 33 lines. Task Elo 1649. Elo 1434 after this task, up 6. Used 3.9k output tokens, 197k in all. Estimated cost $0.122 at list price.

  2. Claude Haiku 5.5Claude Code

    Passed in 1m 42s. 4 reads, 1 edit, 9 test runs, 10 messages. Edited again after a test run once. Changed 23 lines in 1 file; the real fix was 33 lines. Task Elo 1500. Elo 1475 after this task, unchanged. Used 9.6k output tokens, 374k in all. Estimated cost $0.012 at list price.

  3. Claude Opus 5.5Claude Code

    Passed in 2m 5s. 6 reads, 1 edit, 5 commands, 5 test runs, 4 messages. Changed 23 lines in 1 file; the real fix was 33 lines. Task Elo 1351. Elo 1453 after this task, down 5. Used 4.8k output tokens, 412k in all. Estimated cost $0.358 at list price.

Task 2, first run 17 Oct

Each model got the same issue in the same codebase. A change passes when the task's tests pass and nothing that passed before breaks. Real fix: 69 lines.

3 of 3 passed. Best on this task: Claude Sonnet 5.5 and Claude Opus 5.5. Fastest pass: Claude Sonnet 5.5 in 1m 18s. Smallest passing change: Claude Opus 5.5, 53 lines against 69 in the real fix.

  1. Claude Sonnet 5.5Claude Code

    Passed in 1m 18s. 4 reads, 2 edits, 4 test runs, 3 messages. Edited again after a test run once. Changed 56 lines in 1 file; the real fix was 69 lines. Task Elo 1545. Elo 1436 after this task, up 2. Used 4.7k output tokens, 214k in all. Estimated cost $0.124 at list price.

  2. Claude Opus 5.5Claude Code

    Passed in 1m 53s. 4 reads, 2 edits, 6 test runs, 5 messages. Edited again after a test run once. Changed 53 lines in 1 file; the real fix was 69 lines. Task Elo 1545. Elo 1455 after this task, up 2. Used 4.1k output tokens, 267k in all. Estimated cost $0.276 at list price.

  3. Claude Haiku 5.5Claude Code

    Passed in 2m 29s. 9 reads, 1 edit, 1 command, 8 test runs, 16 messages. Changed 59 lines in 1 file; the real fix was 69 lines. Task Elo 1409. Elo 1471 after this task, down 4. Used 17k output tokens, 662k in all. Estimated cost $0.019 at list price.

Task 3, first run 17 Oct

Each model got the same issue in the same codebase. A change passes when the task's tests pass and nothing that passed before breaks. Real fix: 48 lines.

3 of 3 passed. Best on this task: Claude Opus 5.5. Fastest pass: Claude Sonnet 5.5 in 2m 23s. Smallest passing change: Claude Opus 5.5, 32 lines against 48 in the real fix.

  1. Claude Opus 5.5Claude Code

    Passed in 2m 44s. 4 reads, 1 edit, 8 test runs, 5 messages. Edited again after a test run once. Changed 32 lines in 3 files; the real fix was 48 lines. Task Elo 1544. Elo 1456 after this task, up 1. Used 4.8k output tokens, 389k in all. Estimated cost $0.377 at list price.

  2. Claude Sonnet 5.5Claude Code

    Passed in 2m 23s. 7 reads, 2 edits, 8 test runs, 5 messages. Edited again after a test run once. Changed 48 lines in 3 files; the real fix was 48 lines. Task Elo 1500. Elo 1436 after this task, unchanged. Used 6.4k output tokens, 459k in all. Estimated cost $0.212 at list price.

  3. Claude Haiku 5.5Claude Code

    Passed in 3m 8s. 18 reads, 12 edits, 10 commands, 8 test runs, 25 messages. Edited again after a test run once. Changed 48 lines in 3 files; the real fix was 48 lines. Task Elo 1456. Elo 1469 after this task, down 2. Used 22k output tokens, 1.89M in all. Estimated cost $0.036 at list price.

Task 1, first run 21 Oct

Each model got the same issue in the same codebase. A change passes when the task's tests pass and nothing that passed before breaks. Real fix: 124 lines.

2 of 2 passed. Best on this task: Claude Sonnet 5.5 and Claude Opus 5.5. Fastest pass: Claude Sonnet 5.5 in 3m 27s. Smallest passing change: Claude Opus 5.5, 135 lines against 124 in the real fix.

  1. Claude Sonnet 5.5Claude Code

    Passed in 3m 27s. 9 reads, 11 edits, 1 command, 6 test runs, 7 messages. Edited again after a test run once. Changed 218 lines in 3 files; the real fix was 124 lines. Task Elo 1500. Elo 1436 after this task, unchanged. Used 18k output tokens, 1.13M in all. Estimated cost $0.524 at list price.

  2. Claude Opus 5.5Claude Code

    Passed in 4m 55s. 11 reads, 4 edits, 3 commands, 6 test runs, 11 messages. Edited again after a test run once. Changed 135 lines in 3 files; the real fix was 124 lines. Task Elo 1500. Elo 1456 after this task, unchanged. Used 18k output tokens, 1.11M in all. Estimated cost $1.02 at list price.

Task 2, first run 21 Oct

Each model got the same issue in the same codebase. A change passes when the task's tests pass and nothing that passed before breaks. Real fix: 63 lines.

2 of 2 passed. Best on this task: Claude Opus 5.5. Fastest pass: Claude Opus 5.5 in 3m 38s. Smallest passing change: Claude Opus 5.5, 42 lines against 63 in the real fix.

  1. Claude Opus 5.5Claude Code

    Passed in 3m 38s. 15 reads, 9 edits, 2 commands, 9 test runs, 8 messages. Edited again after a test run 2 times. Changed 42 lines in 2 files; the real fix was 63 lines. Task Elo 1560. Elo 1457 after this task, up 1. Used 12k output tokens, 1.06M in all. Estimated cost $0.869 at list price.

  2. Claude Sonnet 5.5Claude Code

    Passed in 4m 35s. 17 reads, 19 edits, 3 commands, 12 test runs, 9 messages. Edited again after a test run 2 times. Changed 48 lines in 2 files; the real fix was 63 lines. Task Elo 1440. Elo 1435 after this task, down 1. Used 18k output tokens, 1.77M in all. Estimated cost $0.660 at list price.

Task 3, first run 21 Oct

Each model got the same issue in the same codebase. A change passes when the task's tests pass and nothing that passed before breaks. Real fix: 92 lines.

1 of 2 passed. Best on this task: Claude Opus 5.5.

  1. Claude Opus 5.5Claude Code

    Passed in 3m 31s. 14 reads, 6 edits, 1 command, 7 test runs, 8 messages. Edited again after a test run 2 times. Changed 96 lines in 3 files; the real fix was 92 lines. Task Elo 1631. Elo 1460 after this task, up 3. Used 11k output tokens, 915k in all. Estimated cost $0.777 at list price.

  2. Claude Sonnet 5.5Claude Code

    Failed in 1m 28s. 18 reads, 1 edit, 1 command, 2 test runs, 4 messages. Changed 87 lines in 3 files; the real fix was 92 lines. 1 test that passed before broke. Task Elo 1369. Elo 1433 after this task, down 2. Used 8.9k output tokens, 973k in all. Estimated cost $0.383 at list price.

Estimated cost is tokens at API list prices, not money spent: the Claude and Codex runs use plan usage. A dash means the run stored no token count.

From the blogLatest reports

All posts

02 · Station reportsEach model against its own past

  • Stable
  • Watch
  • Changed
  • Degraded
ModelStatusTrendRatingChangePass ratelatest graded dayDays
GPT-6 Luna (high)Codex · effort highShow details Building baseline, day 1 of 14 1780 ±184 baseline 100% 1
Claude Opus 5.5Claude Code · effort mediumShow details Building baseline, day 7 of 14 1707 ±197 +48 ±281 vs baseline 100% 7
Claude Haiku 5.5Claude Code · effort defaultShow details Building baseline, day 3 of 14 1657 ±197 +19 ±241 vs baseline 100% 3
DeepSeek V4.1 FlashDeepSeek Harness · effort defaultShow details Building baseline, day 2 of 14 1649 ±200 −48 ±278 vs baseline 100% 2
Grok 4.7Cursor · effort mediumShow details Building baseline, day 3 of 14 1649 ±200 −10 ±283 vs baseline 100% 3
GPT-6 AstraCodex · effort mediumShow details Building baseline, day 5 of 14 1640 ±199 −19 ±282 vs baseline 100% 5
GPT-6.1 SolCodex · effort mediumShow details Building baseline, day 4 of 14 1640 ±199 −113 ±273 vs baseline 100% 4
GPT-6 LunaCodex · effort mediumShow details Building baseline, day 3 of 14 1640 ±199 +220 ±231 vs baseline 100% 3
Claude Sonnet 5.5Claude Code · effort highShow details Building baseline, day 7 of 14 1529 ±163 +115 ±214 vs baseline 67% 7
Kimi K3Kimi Code · effort defaultShow details Building baseline, day 3 of 14 1529 ±229 −6 ±322 vs baseline 100% 3

03 · ReportWhat every run took

Not a leaderboard. Models run at different effort settings and through different tools.

Tasks graded
1453 timeouts · 34 not scored
Time spent
9h 16mover 13 days
Tokens used
198.9M197.2M in · 1.78M out
Estimated cost
$97.55at API list prices

Estimated cost is tokens at API list prices, not money spent: the Claude and Codex runs use plan usage. A dash means no list price is set for that model. Graded counts use passing and failing runs; time, tokens and cost can also include finished runs excluded from grading.

ModelPass rate, all daysTimeoutsMean timeMedian timeOutput tokens / taskThinking shareOutput tokens / sEst. cost / taskEst. cost / day
GPT-6 Luna (high)Codex · effort high 100%12/12 0 2m 56s 2m 46s 9.5k1.71M in 44% 57mean 55 $0.029 $0.352
Claude Opus 5.5Claude Code · effort medium 100%21/21 0 3m 34s 3m 26s 8.7k580k in 22% 43mean 41 $0.564 $1.69
Claude Haiku 5.5Claude Code · effort default 94%17/18 1 4m 42s 3m 32s 29k1.92M in 55% 120mean 121 $0.041 $0.230
DeepSeek V4.1 FlashDeepSeek Harness · effort default 100%9/9 0 5m 26s 5m 23s 39k4.68M in – 116mean 118 $0.049 $0.222
Grok 4.7Cursor · effort medium 89%8/9 1 7m 40s 7m 59s 21k2.50M in – 61mean 53 $1.74 $4.65
GPT-6 AstraCodex · effort medium 100%15/15 0 3m 05s 2m 51s 3.5k723k in 7% 19mean 20 $1.34 $4.03
GPT-6.1 SolCodex · effort medium 100%18/18 0 2m 15s 2m 13s 3.9k707k in 12% 28mean 28 $0.200 $0.901
GPT-6 LunaCodex · effort medium 80%12/15 0 3m 06s 2m 13s 5.3k1.22M in 26% 34mean 37 $0.021 $0.106
Claude Sonnet 5.5Claude Code · effort high 83%20/24 1 2m 54s 1m 41s 9.2k652k in 27% 68mean 73 $0.298 $0.980
Kimi K3Kimi Code · effort default 100%4/4 0 10m 19s 10m 30s 15k3.20M in – 24mean 24 $9.83 $13.10

Time per task

How long each task took. A task not finished in 15 minutes is a fail, and counts as 15 minutes of time spent.

  • GPT-6 Luna (high)median 2m 46sslowest 5m 25s · pull-request
  • Claude Opus 5.5median 3m 26sslowest 7m 01s · pull-request
  • Claude Haiku 5.5median 3m 32sslowest timed out · pull-request
  • DeepSeek V4.1 Flashmedian 5m 23sslowest 8m 46s · pull-request
  • Grok 4.7median 7m 59sslowest timed out · pull-request
  • GPT-6 Astramedian 2m 51sslowest 5m 05s · pull-request
  • GPT-6.1 Solmedian 2m 13sslowest 3m 28s · pull-request
  • GPT-6 Lunamedian 2m 13sslowest 12m 31s · pull-request
  • Claude Sonnet 5.5median 1m 41sslowest timed out · pull-request
  • Kimi K3median 10m 30sslowest 13m 20s · pull-request

By task family

Pass rate, mean time and mean output tokens for each kind of problem.

Modelpull-request
GPT-6 Luna (high)100%2m 56s9.5k
Claude Opus 5.5100%3m 34s8.7k
Claude Haiku 5.594%4m 42s29k
DeepSeek V4.1 Flash100%5m 26s39k
Grok 4.789%7m 40s21k
GPT-6 Astra100%3m 05s3.5k
GPT-6.1 Sol100%2m 15s3.9k
GPT-6 Luna80%3m 06s5.3k
Claude Sonnet 5.583%2m 54s9.2k
Kimi K3100%10m 19s15k

Day by day

GPT-6 Luna (high)

1 day · time spent per day
DayPassedTimeoutsTimeEst. cost
8 Oct12/12035m 17s$0.352

Claude Opus 5.5

7 days · time spent per day
DayPassedTimeoutsTimeEst. cost
21 Oct3/3012m 00s$2.67
17 Oct3/306m 38s$1.01
15 Oct3/3010m 10s$1.16
8 Oct3/308m 31s$1.21
7 Oct3/309m 47s$1.51
5 Oct3/3016m 35s$2.01
4 Oct3/3011m 14s$2.29

Claude Haiku 5.5

3 days · time spent per day
DayPassedTimeoutsTimeEst. cost
17 Oct3/307m 15s$0.067
9 Oct3/308m 21s$0.083
8 Oct11/1211h 09m$0.540

DeepSeek V4.1 Flash

2 days · time spent per day
DayPassedTimeoutsTimeEst. cost
7 Oct3/3012m 07s$0.121
5 Oct6/6036m 47s$0.322

Grok 4.7

3 days · time spent per day
DayPassedTimeoutsTimeEst. cost
7 Oct3/3019m 33s$4.02
5 Oct2/3127m 21s$3.22
4 Oct3/3022m 04s$6.72

GPT-6 Astra

5 days · time spent per day
DayPassedTimeoutsTimeEst. cost
15 Oct3/307m 23s$2.95
8 Oct3/306m 54s$3.62
7 Oct3/308m 37s$3.62
5 Oct3/3011m 07s$5.11
4 Oct3/3012m 10s$4.87

GPT-6.1 Sol

4 days · time spent per day
DayPassedTimeoutsTimeEst. cost
15 Oct3/306m 22s$0.533
9 Oct3/306m 28s$0.463
8 Oct3/307m 34s$0.611
7 Oct9/9019m 58s$2.00

GPT-6 Luna

3 days · time spent per day
DayPassedTimeoutsTimeEst. cost
9 Oct3/305m 28s$0.042
8 Oct3/303m 46s$0.038
7 Oct6/9037m 15s$0.238

Claude Sonnet 5.5

7 days · time spent per day
DayPassedTimeoutsTimeEst. cost
21 Oct2/309m 25s$1.57
17 Oct3/304m 39s$0.458
15 Oct3/303m 26s$0.512
9 Oct3/306m 11s$0.587
8 Oct3/305m 36s$0.628
7 Oct2/306m 46s$0.777
4 Oct4/6133m 41s$2.33

Kimi K3

Last 7 days · 3 graded days total · time spent per day
DayPassedTimeoutsTimeEst. cost
14 Oct0/000.0s–
13 Oct0/000.0s–
12 Oct0/000.0s–
11 Oct0/000.0s–
10 Oct0/000.0s–
9 Oct1/1013m 20s$6.22
8 Oct2/2016m 45s$20.72

04 · MethodHow it works

  1. Fresh tasks every day.Each task is an issue in a large, working codebase. The model has to write the change that resolves it, the way it would for a pull request. Every model is sent each task, and only one run per task counts toward a model's results. A model that has not run a task yet shows no result for it.
  2. Done with the model's own coding tool.The model works in a copy of the code through its coding tool, the way a developer uses it. It can read the code, run the tests and fix its own work. Only the final change counts.
  3. Graded by a program.The tests for the change must pass, and the tests that passed before must still pass. No model judges another model.
  4. Rated like a chess player.A change that passes counts as a win against that task's difficulty, which comes from how every model did on it. The rating comes with an error band. Movement inside the band means nothing.
  5. Compared with itself.The last 7 days are tested against the model's first 14 days.
  6. Two signals.Degraded means the pass rate fell. Changed means the model spends a different number of tokens on the work while the pass rate held. Watch is an early warning: the pass rate dipped, but not enough to call.

How each number is computed

Rating
Fitted from passes and fails against each task's difficulty, like a chess rating. The ± is one standard error. The ± on a change combines the errors of the two ratings it compares, so a change inside it means nothing.
Elo
Every pair of models on a task plays one game. A passing change beats a failing one. When both pass, the game is split over the size of the change (counts double), time and estimated cost, using the measures both runs have. Task Elo is fitted on that task alone, and rows are ordered by it. Elo starts at 1500 and is fitted over every task so far. It is separate from the rating, which compares each model with its own past.
Pass rate
Passes divided by passes plus fails. The model table and its detail card show the latest graded day; the report shows all days. A change passes only when its tests pass and nothing that worked before breaks. Errors and refusals are left out.
Timeout
The change was not finished within 15 minutes. It is graded as a fail and counted as 15 minutes of time spent.
Time per task
Seconds from sending the task to the coding tool finishing. The mean and the median cover every task that finished or timed out.
Tokens
Input, output and thinking tokens as the provider reports them, for answered tasks. Thinking share is thinking tokens over output tokens.
Output tokens per second
A task's output tokens divided by its time, for answered tasks only. The table shows the median and the mean.
Estimated cost
Tokens multiplied by the API list price per million. It is not money spent: the Claude and Codex runs use plan usage. A model with no list price shows a dash.
Task race
Each lane starts when the coding tool was started. A mark sits at the moment the tool reported that step. Runs graded before step times were stored show only their start, their finish and their step count.
Tasks
Issues from a large codebase that is in daily development, each one a change a developer would open a pull request for. Every model gets the same set.

The daily results behind every number are free to download as CSV or JSON: one row per model per day. The tasks, the tests and the code the models wrote are not included.