Literally it’s a black box of statistics. Tokens in, Next token probabilities out
You literally just repeated the exact same mistake, albeit in different words. This is tantamount to speaking of airplanes: "It's not flight, it's just a bunch of fluid mechanics". You're focusing on the internal implementation details, not the expressed behavior.
I don't know if people do this out of some sort of "biological chauvinism" or what, but it strikes me as rather odd.
I can (to pick one random example out of many) ask ChatGPT "Who is the current Prime Minister of Japan and what is the largest prime number smaller than their age?" and get a correct answer. To say that a system that can pull that feat off isn't intelligent boggles the imagination IMO. Token prediction? Who cares. Hallucinations? Humans hallucinate also. Perhaps less frequently, but all that says is that we aren't "all the way there" yet with AI.
LLM is intelligent like a human, book, cat, thermostat, or grasshopper is intelligent.
But it's not a human intelligence, and it's creepy when its creators dress it up to pretend it is. It's an extremely superficial imitation of a human, and it doesn't behave like a human. It's like when a human furry claims to really be a dog or an otherkin.
Humans aren't the only intelligence though. Just because we "dress it up" to, well, interact with us humans doesn't mean it's not intelligent. I mean, what else would you expect them to do, make it emit binary?
"LLM is intelligent like a human, but it's not, and don't pretend it is"
is probably the best summary I've read on the topic
The conspiracist in me believes that they want people to think it’s human like and actually sentient so they can argue that what they’re doing is fair use. It’s just learning like humans learn!
As you said, it boggles imagination, because it feels like chatting with a human. But "having knowledge" does not equal "be intelligent". Would you say a dictionary is intelligent? GPT is just a human-friendly way to access, condensate and present to you information already present on the web. It saves you hours of web search and good old notes taking. That's cool, but that's not intelligence.