I’m tempted to say that ideally, we’d have more in-depth, qualitative reviews of AI models so that it’s possible to form an impression of how suitable each model would be to one’s particular use case (instead of just “this model gets you 34 intelligence per dollar; this one, 38” as if they could be reduced to such numbers), but that would be a lot of effort given the sheer number and frequency of model releases.
I personally would like more reviews of new capabilities unlocked. Like there was a lot of talk when o3 got released that it can do geolocation. Feels more interesting than being better at an existing capability.
AA's intelligence score is based on many benchmarks, you can go look at them all individually if you'd rather focus on specific areas.
Those are still numerical scores and, as far as I know, essentially based on “does it solve the task correctly or not”. By “qualitative”, I meant for example some analysis of how it succeeds (e.g. quality of prose or code) or its failure modes (lack of persistence, etc.).
This version added two private sources which cannot be inspected.