I scratched a similar itch and found local LLMs plus Whisper worked really well to listen in and "DJ" a soundtrack while playing tabletop RPGs with a group. If you want to check it out: https://github.com/sean-public/conductor