>a benchmark where LLMs play StarCraft BW by writing code
Around the time Starcraft 2 came out, I talked about it with a friend who was very enthusiastic about the subject. He said he thought the game was awesome, but it was really too bad that it imposed all this mindless tedium on you. (He was a programmer, of course!)
He said it would be really cool if you could just give higher level directives to your units. As though you were commanding them from far away. (The game, in his vision, would somehow account for speed of light, across vast distances.)
I think he really just wanted to play Civ or something :) But this post reminded me of that conversation.
They call that style an Auto-Battler RTS. "Mechabellum" is the most notable title.
Honestly, I always wanted a game to have both. That is, let you freely zoom in all the way to micro-level of StarCraft, and even further, impersonate arbitrary units of your army for an FPS experience, or zoom out to a Tycoon level, or 4x level, and issue macro orders to your bases and fleets all around the sector of the galaxy, all within a single simulation.