>They speak fluently and confidently about experiences it’s impossible for them to have.
But they're echoing these things from people who really have.
The key is to not forget that LLMs are just next-generation search engines, instead of anthropomorphizing them to be "speaking agents". The natural language IO interface is just a side effect.
Sorry to be annoying on that, but if there's one thing LLMs certainly aren't are search engines. They don't index content predictably, they lossly compress it, and furthermore in a manner that the loss cannot be quantified. If you are using them as more than a semi-random/fuzzy content production engines, you are doing it wrong (you're not alone in that, but that's besides the point)
It is a search engine in the sense that it's returning to you a subset of knowledge from a large body of knowledge stored within it. Natural language is simply the interface by which you make the query and by which it can return its results.
And true, it's not deterministic in our experience, but this is an optimization. You could make it so that one particular prompt always returns the same response every single time; the random jitter is "added in" because it happens to produce better results overall.