So if scores are good, they must be benchmaxxed, and if they are bad, they must not be benchmaxxed. So basically, the narrative is whatever you want it, no matter what the data says.
So if scores are good, they must be benchmaxxed, and if they are bad, they must not be benchmaxxed. So basically, the narrative is whatever you want it, no matter what the data says.
No, if a model scores good on benchmarks it could be that they have benchmaxxed, if a modern model scores bad on benchmarks it could be a really bad model but it almost certainly isn't benchmaxxed. It's relatively easy to make a model do well on particular benchmarks, companies that don't care about that stand a better chance at making a model that does well in general because tweaking a model for benchmarks usually is at the expense of something else.
I run my own benchmarks (this model is - unfortunately too large so that won't be happening here) on data that no outsider has ever seen and this gives sometimes results that are not in line with the order that you would expect if you took the various benchmark sites into account. It is of course entirely possible that our benchmark is not representative but it seems to work quite well as a predictor for which models have value for us and which do not. Recently we looked a model that has been pushed with a significant amount of hype and for the life of me we could not get it to perform even close to the published benchmarks, whereas the models we used for controls were definitely within the margin of error. It's not always equally cut-and-dry.