StarCraft probably has hundreds of bot implementations lying on GitHub. Which makes this bench a recall-from-the-training-set competition, essentially. I tried that method on old games that certainly do not have a ton of bot implementations on GitHub. It revealed some interesting aspects about LLM reasoning and ways to use it. https://replicated.live/blog/games
Overall, this is a useful benchmarking approach. May take some newer papers and evaluate them this way.
When I shared the initial results with the Brood War AI community they also had these concerns about recall-from-the-training-set.
I have shared examples that the tested AI models have written. It turns out that the AI models write code that does not resemble any of the bots in the training data.
https://github.com/llmskirmish/starskirmish-hillclimb