> As humans can be reduced to physics, and physics can be expressed as a computer program
This is an assumption that many physicists disagree with. Roger Penrose, for example.
> As humans can be reduced to physics, and physics can be expressed as a computer program
This is an assumption that many physicists disagree with. Roger Penrose, for example.
That's true, but we should acknowledge that this question is generally regarded as unsettled.
If you accept the conclusion that AGI (as defined in the paper, that is, "solving [...] problems at a level of quality that is at least equivalent to the respective human capabilities") is impossible but human intelligence is possible, then you must accept that the question is settled in favor of Penrose. That's obviously beyond the realm of mathematics.
In other words, the paper can only mathematically prove that AGI is impossible under some assumptions about physics that have nothing to do with mathematics.
> then you must accept that the question is settled in favor of Penrose. That's obviously beyond the realm of mathematics.
Not necessarily. You are assuming (AFAICT) that we 1. have perfect knowledge of physics and 2. have perfect knowledge of how humans map to physics. I don't believe either of those is true though. Particularly 1 appears to be very obviously false, otherwise what are all those theoretical physicists even doing?
I think what the paper is showing is better characterized as a mathematical proof about a particular algorithm (or perhaps class of algorithms). It's similar to proving that the halting problem is unsolvable under some (at least seemingly) reasonable set of assumptions but then you turn around and someone has a heuristic that works quite well most of the time.
Where am I assuming that we have perfect knowledge of physics?
To make it plain, I'll break the argument in two parts:
(a) if AGI is impossible but humans are intelligent, then it must be the case that human behavior can't be explained algorithmically (that last part is Penrose's position).
(b) the statement that human behavior can't be explained algorithmically is about physics, not mathematics.
I hope it's clear that neither (a) or (b) require perfect knowledge of physics, but just in case:
(a) is true by reductio ad absurdum: if human behavior can be explained algorithmically, then an algorithm must be able to simulate it, and so AGI is possible.
(b) is true because humans exist in nature, and physics (not mathematics) is the science that deals with nature.
So where is the assumption that we have perfect knowledge of physics?
You didn't. I confused something but looking at the comment chain now I can't figure out what. I'd say we're actually in perfect agreement.
"Many" is doing a lot of work here.
Penrose’s views on consciousness is largely considered quackery by other physicists.