Daily test results · Codex · effort medium
Is GPT-6.1 Sol getting worse?
No verdict yet: day 4 of a 14-day baseline.
- Runs through
- Codex
- Effort
- medium
- Provider
- codex-cli
- Model id
gpt-6.1-sol- Tasks
- New issues in real code
- Updated
- 2026-10-21 04:41 UTC
Rating over time
Higher is better. The band is one standard error either side. The dashed line is the first day. Movement inside the band means nothing.
Was GPT-6.1 Sol nerfed?
No verdict yet
This is day 4 of the 14-day baseline for GPT-6.1 Sol. The baseline began on 7 October 2026. If the model runs every day, the baseline is complete on 25 October 2026 and the first verdict is expected on 26 October 2026.
It is too early to say either way. Until the baseline is complete there is nothing to compare a new day with, and a single day moves by chance.
- Rating
- 1640±199 · one standard error
- Change since day 1
- −113from 1753 on 7 October 2026
- Pass rate, latest graded day
- 100%latest graded day · 3 graded tasks
- Days of data
- 4since 7 October 2026
What the runs took
- Median time per task
- 2m 13smean 2m 15s · 0 timeouts
- Output tokens per task
- 3.9k12% thinking · 707k in
- Output tokens per second
- 28median · mean 28
- Estimated cost per task
- $0.200at API list prices · $0.901 a day
Estimated cost is tokens at API list prices, not money spent. A dash means no list price is set. Graded counts cover 18 tasks over 4 days; time, tokens and cost can also include finished runs excluded from grading.
Time per task
How long each task took. A task not finished in 15 minutes is a fail, and counts as 15 minutes of time spent.
By task family
Pass rate, mean time and mean output tokens for each kind of problem.
| Model | pull-request |
|---|---|
| GPT-6.1 Sol | 100%2m 15s3.9k |
Day by day
| Day | Rating | Passed | Timeouts | Time spent | Median time | Output tokens | Output tokens / s | Est. cost |
|---|---|---|---|---|---|---|---|---|
| 15 Oct | 1640 ±199 | 3/3 | 0 | 6m 22s | 2m 13s | 8.5k | 24 | $0.533 |
| 9 Oct | 1640 ±199 | 3/3 | 0 | 6m 28s | 1m 53s | 9.8k | 25 | $0.463 |
| 8 Oct | 1611 ±203 | 3/3 | 0 | 7m 34s | 2m 04s | 12k | 26 | $0.611 |
| 7 Oct | 1753 ±187 | 9/9 | 0 | 19m 58s | 2m 22s | 39k | 32 | $2.00 |
How this is measured
GPT-6.1 Sol gets the same new tasks as every other model. Each task is an issue in a large, working codebase, and the model writes the change that resolves it with its own coding tool, as it would for a pull request. A program runs the tests to check the change. Only one run per task counts toward this model's results.
The model is compared with itself: the last 7 days against its first 14 days. A verdict is only given when the difference is too large to be chance. A task not finished in 15 minutes counts as a fail.
Not a leaderboard. Models run at different effort settings and through different tools.
Other models
- GPT-6 Luna (high)Codex · effort high
- Claude Opus 5.5Claude Code · effort medium
- Claude Haiku 5.5Claude Code · effort default
- DeepSeek V4.1 FlashDeepSeek Harness · effort default
- Grok 4.7Cursor · effort medium
- GPT-6 AstraCodex · effort medium
- GPT-6 LunaCodex · effort medium
- Claude Sonnet 5.5Claude Code · effort high
- Kimi K3Kimi Code · effort default