I wanted the same, so I hacked/bolted on an LLM to Morrowind (the open source recreation OpenMW).
The biggest problem I faced at the time (during ChatGPT 3 era) was that, without a good context, LLMs are the most vanilla roleplayers you’ve ever seen. By themselves, LLMs are just not interesting enough for a player to choose to talk to in-game.
If you want them to be “interesting” to talk to, you must provide (or generate and keep track of): a backstory, chat history, the scene, NPC inventory, the NPC’s current emotional state, the weather, literally everything needs to be given to the model before it generates messages for the player.
At which point you’ve got a big task. You need a way to automatically get the relevant data to the model for the specific conversation you’re having. There might be tools to pick appropriate text documents from a db given a conversation topic, but I didn’t/don’t know how to make that work for games.
I’m sure there’s a way to accomplish this with more modern tools and models. (Maybe instead of providing all that data up front, you would now give the model tools to call to retrieve that data on-demand?) But that’s what made me give up in 2022.
Of course it would require the game engine to provide context to the LLM, it certainly is quite a bit of work but nothing technically impossible and the result are endless possibilities and endless replayabilities. You could very well let the LLM decide if an npc should agree to give an object to the player or not, even let the player try to convince the npc.
I mean it looks like to me the next big step in gaming after 3D and yet this is being ignored.