I still call them stochastic parrots, but believe what they are revealing is that we are all stochastic parrots to some extent. I simply don't see how biological computation (i.e. thinking) can be anything else. Similar to the reveal in west world, we are likely much simpler than we give ourselves credit for.
A "train of thought" can be seen as a trace of a depth first search where the preceding trace is used to guide termination and next expansion decisions. A similar concept, "taboo search", exists in classical constraint optimization where previous solutions are fit to a model that guides future expansion (but as the name "taboo" implies, away from uninteresting solutions).
We also have harnesses that perform breath first search.
If I tried to describe what it means to "think deeply", I would probably say a combination of both.
Ultimately I believe that we will surpass human capabilities but fail with alignment. Handing the world's resources over to stochastic systems that can evolve faster than we can reason about them simply leaves too many "interesting" outcomes that do not end well. I also expect the failure modes will be totally non-obvious.
> we are all stochastic parrots to some extent.
I think this statement is continuation of the old fallacy - every generation thinks of brain in terms of what is the current technology zaitgeist is - was it 19th century when they thought brain is a network of pneumatic pipes?
As long as there's enough of them with different goals it doesn't matter, they'll keep each other in check. The worlds resources are already handed over to the worst people and we're still doing fine and none of the billionaires are "aligned with society". They just align with their own belly but because they want different things it all kinda works.
These “worst people” need you. They physically need you alive to perform labor for them and to give them money (and status).
That’s the reason we are “doing fine”. Once they stop needing you..
Also, both our comments brush over the generational struggles for fairness over the centuries. We have fought to be “fine”, it did not just happen. Without fairness being introduced by force you and I would be slaving away in some sweatshop getting paid nickels as was the norm not so long ago.
Edit: That’s also assuming you are Caucasian. If you are of a different ethnicity.. well, historically, all bets are off. You could also be the literal possession of some of these “worst people” with not even your own children considered yours.
My only point is in a many agent system with different goals it "doesn't matter" that some agents have bad goals as long as there's enough variability of goals and resources that they can't put their vision in place.