Clearly nature avoids this problem. So theoretically by replicating natural selection or something else in AI models, which arguably we already do, the theoretical entropy trap clearly can be avoided, we aren't even potentially decreasing entropy with AI training since doing so uses power generation which increases entropy
If we did that, would we be really replicating what nature does, or would we be just simulating it?
Human intelligence and consciousness are embodied. They are emerging features of complex biological systems that evolved over thousands and millions of years. The desirable intelligent behaviours that we seek to replicate are exhibited by those same biological systems only after decades of growth and training.
We can only hope to simulate these processes, not replicate them exactly. And the problem with such a simulation is that we have no idea if the stuff that we are necessarily leaving out is actually essential to the outcome that we seek.
It doesn't matter wrt the claims the article makes, though. If AGI is an emergent feature of complex biological systems, then it's still fundamentally possible to simulate it given sufficient understanding of said systems (or perhaps physics if that turns out to be easier to grok in full) and sufficient compute.
It can be avoided certainly, but can it be avoided with the current or near term technology about which many are saying “it’s only a matter of time”
I like the distinction you made there. My observation that when it comes to AGI, there are those who are saying "Not possible with the current technology." and "Not possible at all, because humans have [insert some characteristic here about self awareness, true creativity, etc] and machines don't.
I can respect the first argument. I personally don't see any reason to believe AGI is impossible, but I also don't see evidence that it is possible with the current (very impressive) technology. We may never build an AGI in my lifetime, maybe not ever, but that doesn't mean it's not possible.
But the second argument, that humans do something machines aren't capable of always falls flat to me for lack of evidence. If we're going to dismiss the possibility of something, we shouldn't do it without evidence. We don't have a full model of human intelligence, so I think it's premature to assume we know what isn't possible. All the evidence we have is that humans are biological machines, everything follows the laws of physics, and yet, here we are. There isn't evidence that anything else is going on other than physical phenomenon, and there isn't any physical evidence that a biological machine can't be emulated.