Ok my original comment was snarky (apologies to all three mods), but you've raised one of my favorite topics so I can't resist engaging a bit more seriously! In the second clause, that is -- the first is trivially true in some senses and trivially false in others, so it doesn't seem very productive to nail down.
So Chomsky thought of himself as a philosopher of science more than anything, and his life mission is/was to answer a fascinating question: what kind of creatures are we? There's been a growing consensus going back to at least Darwin that we're separated from the rest of Animalia by our "reason" (!= intuition != understanding != judgement != volition), but he helped popularize an old theory[1] that puts it kind of beautifully: human language is only possible in beings capable of producing an infinite set of meaningful outputs in response to a finite input.[2] And so far, that list includes only two definite members: Homo sapiens sapiens, and computer programs built upon large language models.[3]
So all that to say: yes, I think you're wrong; the programmatic ensembles that brought us today's breakthroughs are an "infinitely deep well". As long as we have questions to ask, they are not theoretically constrained from answering them.
Of course, this invites some obvious follow-ups on practicality; even the most impassioned ASI boomer wouldn't argue that the exact ensembles used here could answer every question in the universe. But A) that's possible because some infinities are bigger than others, and B) the exact same caveat applies to humans, whose cognitive capabilities are thoroughly determined by accidents of biology, culture, and episteme. The latter is the thrust of Chomsky's famous debate with Foucault, which is absolutely worth a watch for anyone who hasn't seen it (starting around ~18:00, AFAIR): https://youtu.be/3wfNl2L0Gf8?t=1080
---
TL;DR: I ultimately disagree on both points, yes. In the ways that matter, machines are no more constrained than animals.
[1]: https://en.wikipedia.org/wiki/Wilhelm_von_Humboldt
[2]: Many animals are capable of communication, often quite complex and fascinating communication at that; even flora and fungi communicate! But language is qualitatively distinct. You may have heard this in the pithy-yet-mindbending sentiment "we've taught many apes to communicate with sign language, but none of them have ever asked a question."
[3]: It's no surprise in hindsight that trying to build better autocorrect got us to AGI; I find it unspeakably tragic that Chomsky never got the chance to see how thoroughly he's been vindicated, in this regard. He had his stroke mere months into the singularity, and had a quite negative view of the approach before then.
> human language is only possible in beings capable of producing an infinite set of meaningful outputs in response to a finite input. > And so far, that list includes only two definite members: Homo sapiens sapiens, and computer programs built upon large language models.
In LLMs that's a bounded infinity though, the models themselves have only so many bits and the activations are also limited. Once you reach the same state twice the model is looping and it won't break out of that loop. I've seen such loops across tens of thousands of tokens so I'm not sure to what degree the infinite output rule holds true. It would be an interesting experiment to try to figure out to what extent the 'busy beaver' can be kept busy before it degenerates to looping or nonsense.
Humans are finite bodies.
LLMs are just regurgitating an output according to a set of rules generated from harvesting all human output.
Language to humans is a symbolic mapping of labels to reality based on actual experience.
"Language" to LLMs is just a string of tokens to manipulate based on relationships harvested from recorded human experience.
It doesn't mean anything other than 1s and 0s - it's not a language because the LLM isn't communicating with you, just as the sum of humanity isn't communicating with you.
It's just a really good "choose your own adventure" book.
...do you think molecules and their interactions are also not bounded physical systems? What part of a human being do you think isn't a turing computer? Please release your groundbreaking physics paper proving this.
Yes they are.
Humans aren't magical.
They also aren't a glorified library. They are authors, they generate the books in the library.
Cross reference a load of books and you will get useful relational info, but it's still a library index in a library.
Apollo the African grey parrot not infrequently asks “what this?”
He may just be parroting something he heard.