Daily test results · Claude Code · effort high
Is Claude Sonnet 5.5 getting worse?
No verdict yet: day 2 of a 14-day baseline.
- Runs through
- Claude Code
- Effort
- high
- Provider
- claude-cli
- Model id
claude-sonnet-5-5- Tasks
- 30 new ones every day
- Updated
- 2026-10-04 05:45 UTC
Rating over time
Higher is better. The band is one standard error either side. The dashed line is the first day. Movement inside the band means nothing.
Was Claude Sonnet 5.5 nerfed?
No verdict yet
This is day 2 of the 14-day baseline for Claude Sonnet 5.5. The baseline began on 3 October 2026. If the model runs every day, the baseline is complete on 16 October 2026 and the first verdict is expected on 17 October 2026.
It is too early to say either way. Until the baseline is complete there is nothing to compare a new day with, and a single day moves by chance.
- Rating
- 1466±70 · one standard error
- Change since day 1
- −91from 1557 on 3 October 2026
- Pass rate
- 57%latest day · 30 graded tasks
- Days of data
- 2since 3 October 2026
What the runs took
- Median time per task
- 1m 16smean 2m 47s · 11 timeouts
- Output tokens per task
- 15k96% thinking · 1.1k in
- Output tokens per second
- 129median · mean 138
- Estimated cost per task
- $0.150at API list prices · $3.68 a day
Estimated cost is tokens at API list prices, not money spent. A dash means no list price is set. Numbers cover 60 graded tasks over 2 days.
Time per task
How long each task took. A task with no answer in 8 minutes is a fail, and counts as 8 minutes of time spent.
By task family
Pass rate, mean time and mean output tokens for each kind of problem.
| Model | assign | orders | paths | queens | tsp |
|---|---|---|---|---|---|
| Claude Sonnet 5.5 | 42%6m 10s48k | 92%1m 55s15k | 83%1m 56s10k | 25%3m 31s13k | 67%26s3.2k |
Day by day
| Day | Rating | Passed | Timeouts | Time spent | Median time | Output tokens | Output tokens / s | Est. cost |
|---|---|---|---|---|---|---|---|---|
| 4 Oct | 1466 ±70 | 17/30 | 5 | 1h 12m | 1m 06s | 304k | 127 | $3.09 |
| 3 Oct | 1557 ±76 | 20/30 | 6 | 1h 34m | 2m 05s | 421k | 129 | $4.27 |
How this is measured
Claude Sonnet 5.5 gets 30 new exact-answer reasoning problems every day, at a fixed difficulty mix. No problem is ever sent twice, so there is nothing to memorise. A program grades each answer as right or wrong.
The model is compared with itself: the last 7 days against its first 14 days, at the same difficulty. A verdict is only given when the difference is too large to be chance. A task with no answer in 8 minutes counts as a fail.
Not a leaderboard. Models run at different effort settings and through different tools, and the difficulty mix was tuned to Claude Opus.