So the measure of a model is how well they can recreate something they easily have thousands of examples of in their training data. There's probably a better base RTS on github somewhere for free.
So the measure of a model is how well they can recreate something they easily have thousands of examples of in their training data. There's probably a better base RTS on github somewhere for free.
Well, it is a silly test, not a scientific benchmark.
However, I would say it is a measure (not the measure). If you look at the entries, there's a lot of variation - definitely not something they memorized outright.
And the test itself is deceptively simple. You need to do canvas rendering, there's pathfinding, command queueing, terrain generation, etc. There are some subtle click handler bugs (various LLMs often stumble on those). And I ask the model to do it all in one file, further increasing the complexity of the task.
And the result is something that you can instantly evaluate. And if the result is any good, even play! So yeah, I think it's a fair test.
I'm sure it'll get saturated at some point. Actually I started with Minesweeper and switched to RTS last December, because Minesweeper was being saturated. I'm expecting (hoping?) the RTS test will last until the end of this year...
Ask it to generate a completely bananas game idea and it will easily do it. Prototyping any type of game with simple graphics has been solved since many generations of models back. Claiming it's because it has that game in its training data is nonsense.
The goalpost velocity is approaching light speed…
>There's probably a better base RTS on github somewhere for free.
I... I think you are missing the point.