I wish AA benchmarked the models at the reasoning efforts people actual use them at. For example, it's very rare for people that I know to use the "max" reasoning setting - specially because Claude Code itself scares you away from using it. But all of these coding benchmarks are based on just that reasoning effort.
I get it, they can only benchmark so much. But I'd rather have a benchmark of how the model is actually used and offered to customers rather than the best case scenario for it.
Maybe you need to enable them in the filters, but AA is the best website I found with benchmarks numbers on all reasoning effort levels of recent frontier models.
I dug into it more, and it seems like the reasoning efforts are available for the Intelligence Index, but not the Coding Agents benchmark I was interested in.
Is it different between major models how they scale with reasoning level? Some models where it hardly pays off? I expect it to be very similar across the board.
It is variable, why would independent implementations be highly similar?
The other reasoning levels are almost always benchmarked, AA just doesnt show them in the default charts.
They don’t benchmark different reasoning levels for Deepseek models.
I see them for GPT 6.1 Terra, but not Opus 5.5. I'm looking on the Coding Agents page.
they do