Without knowing what makes consciousness possible, the paper cannot justify biology as necessary - it mistakes a lack of evidence for conscious AI for proof that conscious AI is impossible.
I do not believe current AI or LLMs are conscious, but there is no proof one way or another that they can or cannot be. The paper authors are making up their own definitions and building an argument from them
Your argument requires that there is some objective truth for what consciousness is. It will always hinge on what definition one accepts.
I, and apparently many others, don’t think it would be any useful to describe the mathematical properties of an AI as consciousness. To me it is inherently a way to describe the “experience” arising from physical processes in biological beings as ourselves.
That’s what the argument comes down to for me. Could an LLM “fall unconscious”?
I think you, and apparently many others, are hiding behind mathematical strictures to avoid the discussion. Your argument is actually "what requires an objective truth" Or else you could admit there's no real difference between zapping amino acids with electricity and zapping silicon with electricity.
Why are AIs mathematical beings with no possibility of consciousness whereas humans fall outside this and have the capability of consciousness?