I am not sure I like this benchmark, but mostly, because I am not sure what it is really evaluating. It is not an idle dismissal.. multiple factors seem to complicate what it is supposed to look at ( just top of my head ):
- code generation speed - logic analysis - understanding of the game platform and how use of C may influence it - considered approaches, tactics and strategies
Don't get me wrong. It is is neat idea. I am just not sure what it measures.
Yeah giving a 1 hour limit for code generation makes the whole thing a bit incoherent. Plus, performance is just going to depend on how closely the models have memorized the best open-source Starcraft bots and can reproduce them.
> Plus, performance is just going to depend on how closely the models have memorized the best open-source Starcraft bots and can reproduce them.
If that's the case then these models are failing extremely hard. 7% win rate against human written AIs that they should be copying from memory? A copy paste command should do 50%!
If Astra is leading the pack...Considering it's supposed to be one of the slowest model.
> I am just not sure what it measures.
All of the above, to the extent that they can be measured from a simple to verify end condition (stats, win rates, etc). Adaptability, ability to understand and progress on a task, and so on.
Yes, but I guess my objection is that if it attempts to measure everything then we can't really tell what it measures in what proportions.
Yep, that’s a feature of end to end measures, like company revenues, vehicle fuel efficiency, or human life expectancy.
The customer may not care what the internal contributions are to such a measure. Instead it represents the customers’ overall values in some way.
Ok. I think you convinced me based on those. I withdraw the objection.
Also: given that the human written bots are almost certainly in the training data, the winning rate against human-written bot seems to be not well-calibrated. You would expect at least they are 50% or better, given that they can "memorize" the best human-written bot.
Worth digging into the protocol and see what went wrong.
Top bots are in the ballpark of 50kloc. Even large LLMs are not so large that they can memorize all the world's data like that. And all the fine-tuning that compression loses comprises most of the work in the first place.
general "agenticness"