I believe we should treat AA results as just a data point and not the overarching 'authority' they may intend to be.
If we had access to neutral raw benchkmark data in a useful way to compare apples to apples, then we could each define our own 'index' to get the slice of perspective we find useful.
Trying to define or compare 'intelligence' through a single index will always be off.
Such a "neutral raw benchmark" would need to be kept private and highly secured. If the labs get a hold of it, they may benchmaxx it to death
So you really need to develop it yourself. If you buy such benchmark from someone, and the benchmark is any good, you may bet the labs will be buying too
> then we could each define our own 'index' to get the slice of perspective we find useful.
its too much work for 98% AI consumers. In reality, AA is best what we have as aggregated score..
Doubt 98% people care, but for those that want to understand I don't think their main index has a good signal anymore.
> main index has a good signal anymore
maybe (this could be discussed), but the problem there is nothing better..