If you haven't been keeping an eye on LLMs playing games, GPT 6.1 Sol can new beat games 100% vision-only (without harnesses that give the agent extra information about the game). Pokemon is no longer a difficult challenge for it, even with additional difficulty restrictions such as hard mode and level caps. Recently, GPT 6.1 Sol beat Final Fantasy VII and its bonus bosses vision-only surprisingly quickly.
I suspect Opus 5.5 can do even better but there aren't as many people testing it.
Where are the recordings of these games? I know this kind of stuff must be happening, but it seems oddly difficult to track down the actual recordings, which feel like the really fascinating thing people should be publishing if they’re putting in the time and tokens to do stuff like this.
Like I want to see what Pokemon strategies it uses: Does it try to brute force with its starter? Does it try to load all of its HMs onto one or two Pokemon? Does it use Flash? If it needs to catch a new Pokemon to beat a gym leader, how much does it remember about which species it encountered where and what their levels were?
The LLMs no longer brute force a starter, and they have to play competently, including the use of setup. Here's a screenshot of the completion of the GPT-6 Astra run in 21 hours with comically underleveled Pokemon (and it actually savescummed RNG!): https://www.reddit.com/r/ClaudePlaysPokemon/comments/1w79vim...
Another run with Ironlocke rulesets for Crystal (hard mode, level caps, permadeath) has even more diverse teams: https://www.reddit.com/r/ClaudePlaysPokemon/comments/1wf6zx0...
It has been a few years now. By now you should have realized that AI is for doing stuff at a massive x100 productivity boost and never getting a complete project done. (For some reason there's those "final kinks to iron out", once the AI figures out where it fucked up. Just two more prompt modifications.)
If you reading every wiki, forum, dev blog/patch note, watching every video, etc before playing, I’d be more surprised if you could NOT play a game.
Exactly. All of these examples are well documented games. And even then it’s only really good at games that don’t need realtime.
You could make the same argument for these LLMs years ago but there's a reason that they've become faster and more accurate at beating these games. GPT 5.5 took 218 hours to beat Pokemon Fire Red; Astra took 21.
Only months ago did these LLMs make it past the Victory Road boulder puzzle without harness assistance.
GPT-4 was trained on all that too and couldn't get any task at all done
I would be willing to bet it can beat games un-released at training time as long as latency is not a requirement, which does limit the game types a bit.
How do they feed the game screen to the models? I imagine the models are too slow if you send 60FPS, but I feel like that’s also the case if you maybe feed it 1FPS? If you go any slower, you’ll miss too much info?
I would assume it's pull-based so the agent takes a screenshot whenever it finishes a turn.
Essentially yes. For games with some real-time component (FFVII), the game pauses after actions are executed to take a screenshot and let the agent think.
Your last sentence mirrors my model choices recently. Opus can do things very well but Sol can do them well for less than 1/4 the price.
And 10x the time.
Sadly this.
Sol, write me a function that moves all the toothpicks from jar 1 to jar 2
>computing requirements
>planning
>writing test harnesses
>writing tests
>450/450 offline tests pass
>1 toothpick moved...
> verifying
>writing acceptance tests for toothpick 1
>error in acceptance test, fixing...
>toothpick is one pixel from expected destination, restarting toothpick job
>...
>THIS MODEL IS AT CAPACITY. PLEASE TRY ANOTHER MODEL.
It's not a mystery, artificial intelligence ranks opus exactly as 1/8 to 1/4th cheaper.
Opus and claude models have always been lagards in vision. It just hasn't been a focus like coding for anygropic. Gemini and ChatGPT have always been much stronger in this area
What's "anygropic"?
Close on the keyboard to and small Levenshtein distance from Anthropic
typo for Anthropic
I would like to see it play Slay the Spire (or 2) and beet it consistently. Mainly to see what strats it uses. I suspect it wouldn't have too much trouble.
Some of these Games are definitely in the RL environments.
Do you have link for the VII bonus boss run?
The channel is here, with past broadcasts: https://www.twitch.tv/gpt_plays_games