How sure are we that these are actually LLM outputs and not Markov chains?

What’s the difference?

I mean, LLMs are Markov models so their output is a Markov chain?