I am absolutely serious. I agree with you that the extent to which we don’t understand brains is overpowering. And I would stand by the proposition that “we don’t understand LLMs in almost exactly the way we don’t understand brains”.
We know lots about human development and genetics and biology and evolution and neuroscience and the physics of how brains are connected and send signals and how generally they are put together and have names for their parts and all that, but we’re clueless when it comes to “the hard question” of how qualia and consciousness emerges from that.
The scenario with the spooky simulation of thinking that emerges from LLMs is in the same category, with different details. Lots of knowledge about the substrate of the phenomenon, little to none about the much bigger question of how we get the appearance of cognition from these trained artifacts.
Clearly we understand extremely well how LLMs are created mechanically. We invented them and are currently putting massive amounts of work into studying and improving them. But that work is perforce largely empirical; figuring out the why once again eludes us. It just goes to show how mysterious the underlying phenomenon of cognition is.
I'm not following you here, you seem to be conflating LLMs ability of language use with the brain's ability of thinking and cognition?
Are you saying that thinking and cognition requires language use? Cause I think not.
Or are you saying that language use is sufficient for cognition and thinking? Cause I'm also not convinced of that.
What I am convinced of, is that a machine capable of language use is capable of tricking people into believing there's a "there", there. In pretty much the same way as the famous supra-normal stimuli experiment made baby seagulls believe that a stick with a red dot was their parent. It's exploiting our instincts.