I don't trust these Artificial Analysis indexes and I'm not sure why anyone considers them authoritative, the site just sprung up a few months ago and now everyone quotes it?
From my own use of the models I find their ranking to be totally inaccurate.
They are the only site that really combines all benchmarks into a single place. And their intelligence index is pretty broad, including knowledge work and finance tasks as well as coding, so if your use case is narrow it may not match the general benchmarks
Personally I do trust them, but I mainly look at the Intelligence Index vs. Time per Task, and Intelligence Index vs. Cost per Task
The per token charts do not provide meaningful information for me
I also find the provider benchmarks fascinating: https://artificialanalysis.ai/models/deepseek-v4-1-flash-101...
Do you have an alternative? Its a benchmark collection. When a benchmark gets saturated, they remove it from the index and add new ones.
I think lm-arena is a better concept that reflects the actual target users, but it only captures chat, and relies on users spending their time improving it.
https://epoch.ai/eci?view=graph&tab=release-date is at least an alternative. They are slower to update, but entirely anecdotally I find the numbers are aligned with my personal observations a bit better than AA. But I am also highly aware this is some pretty confirmation-bias heavy ares of meta-benchmark picking...
If we ignore the older releases (sonnet 5, etc that exist in that one) and the new models epoch has not benchmarked, we end up with the same order within the noise brackets as in ArtificialAnalysis.
I do trust epoch more than AA due to capital incentives, but in this case I don't think like there is a difference
It's appealing to me because it's not just one benchmark but a summary of a collection of benchmarks. Also the current rankings have Opus on top by far and everyone's experience is that Opus is best by far so it's not massively inaccurate
> From my own use of the models I find their ranking to be totally inaccurate.
It's fine if you don't trust them. You are free to put together your own benchmarks, run your own tests across models, and objectively compare results.
That's how you gathered your data to support your claim other benchmarks are totally inaccurate, right? Right?