> When the consciousness itself not understood and well defined in the first place, it is pretty pointless to debate if something is or isn't conscious.

But we know what consciousness is, even if you don't know how to explain it in materialistic[0] terms. Intentionality is what makes consciousness consciousness. We know that LLMs - and computers in general - do not possess intentionality, not even sensory intentionality. There is no "aboutness" and no semantics to the content within a machine, never mind that the processes within LLMs are not reasoning or inferential processes. It is a sophisticated behavioral simulation that may be syntactically defined, nothing more. It is purely a matter of transeunt causes and not the immanent causality needed for something like consciousness.

(There's also the more general problem that we cannot say that physical computing machines are objectively computing. There is nothing about the physical processes - which completely define what a computing device is doing - that could be identified with computation in any objective sense. Rather, we human observers assign a computational interpretation to the machine's operations, just as we assign them to the ink blobs in a book or the liquid crystals you are reading on your screen now. Computation is something we do, and we have built machines to simulate it. We formalized computation - which is a process of desemantification and syntactic codification - and build machines guided by these formalisms. Formalization is what makes mechanization possible.

> Decomposing the complex activity into simple steps like 'predicting the next word' and claiming that surely can't have consciousness. A similar argument would be -- there is no way that movements of electrons by tiny distance would produce consciousness.

The movement of electrons as such cannot produce consciousness. The reductionist paints himself into a corner by strapping reality into a Procrustean mechanistic and materialistic metaphysics that redefines reality according to some dedicated schema, and then strains endlessly to find how some phenomenon can arise or emerge from the paltry remains. (Either that, or he embraces insanity completely and becomes an eliminativist.) It is magical thinking. I suspect this fallacious line of thinking is what causes people to believe piling more syntactic operations will somehow magically fuse into semantics and intentionality.

[0] "Materialism" here is the metaphysical theory. I am not claiming that consciousness - at least not all consciousness - cannot be a physical phenomenon. I am only claiming that the materialistic view of matter cannot account for consciousness.