I currently operate under the assumption that humans are at most as powerful as Turing Machines. And from what I understand these models internally are modeling increasingly harder and larger DFAs, so they're at least as powerful as regular languages.
Assuming humans are more powerful than regular languages I could maybe agree that these methods may not eventually yield entirely human like intelligence, but just better and better approximations.
The vibe I get though is that we aren't more powerful than regular languages, cause human beings feel computationally bounded. So I could see given enough "human signal" these things could learn to imitate us precisely.
Well yeah there is likely an equivalence between computability and epistemology, but I'm not sure it matters when comparing LLM intelligence to human intelligence. There is clearly a missing link that prevents the LLM from reaching beyond its training data the way humans do.
If you look at the life efforts and accomplishments of the ~100 billion humans who have ever lived, how many lifetimes would you discount as having "non-human intelligence" based on the lack of "novel" contributions to frontier of our species' scientific understanding according to the same high bar you apply to LLMs?
Do you pass that bar yourself?
Ordinary humans do novel things all the time. Where do you think LLMs got all the training data that their responses come from?
You're not quite addressing the question. More and more of the training data is now synthetic.
To be very specific - what novel things did the majority of the ~8 bil humans on Earth do say, yesterday, that you wouldn't otherwise dismiss as non-intelligent rehashing of the same tired patterns they always inhabit were those same actions attributed to LLMs?
What I'm getting at is that I think you're falling into the trap of thinking of the rare geniuses of human history, and furthermore their rare moments of accomplishment (relative to the long span of their lifetimes filled mostly without these accomplishments) when you think of "human intelligence", which is of course far overstating what actual human intelligence is.