>the reasoning behind the argument is bizarre

>Decomposing the complex activity into simple steps like 'predicting the next word' and claiming that surely can't have consciousness

I agree. I think the whole point of natural "intelligence" is to predict future events in order to properly plan actions for survival. The only difference is that next-word prediction happens in a different space (in a non-physical, textual form). But I don't think the distinction matters that much, because by the time the signals reach our brain's "intelligence core," they're already preprocessed multiple times. We can't see real physical reality, we "hallucinate" colors, touch, etc. (I think schizophrenia occurs when this kind of "controlled hallucination" goes wonky, but I may be wrong). So we're not that different from LLMs here.

I'd define consciousness as meta-intelligence, i.e. the ability to reflect on why/how a prediction was made (and make corrections to the pipeline). LLMs so far cannot properly explain why a certain prediction took place, but I'm not sure humans can fully either! I remember there was research showing that by scanning the brain (or some other signals), you can predict a person's choice before they're even aware of it. It's possible that our explanations are post hoc as well, and that the meta-cognitive ability to explain our own reasoning is as rudimentary as LLMs' (see: all the biases).

If you think about it this way, the only difference left is that everything we do is based on survival. LLMs don't have this goal. But I'm not sure it's relevant to the concept of consciousness.

A few months ago, I recreated Qwen3's architecture in 30 lines of code, and it gave me a sort of existential crisis: does it really take just 30 lines of code and a float array to recreate something that thinks and sounds almost like me? Is that all there is to it? There's often the argument that the brain is much more complex than 30 lines of code, which is true, but in my opinion, a lot of brain structures are basically archaic legacy systems (or auxiliary subsystems) that are not strictly necessary for human intelligence (which is bolted onto those legacy systems). If you carefully remove 95% of the brain (if you know where to remove), you can probably still have consciousness left in some form, it just wouldn't be very capable of surviving on its own.

I think our ability to debate consciousness in machine learning systems is greatly hindered by our subconscious existential fear: the fear that we ourselves can be reduced to non-conscious bits and simple mechanisms. In a way, it feels like a destruction of the ego.