We used to use markov chain generators of various types to spit out absolute insanity, in the early years of the century, and this is all I ever wanted from LLMs. I bet if you used a slightly older model for your text generation, you'd get really clean output. Sonnet 4.whatever, I think, actually does a superb job at text generation, despite it being only an ok coding model.