We humans have the habit of falling into the trap of anthropocentric views and explanations of the universe and everything within it. The history of AI is the history of moving goalposts. I don't particularly care if they are conscious or not. Humans have a history of treating even actual, conscious, intelligent beings as chattel or food. So I don't really see why our attitudes would change if we somehow realized that LLMs are conscious.
I have been thinking about this for a while in the context of the quest for AGI. If all we want is something to take over some parts of our intellectual labor, then LLMs are already there. They can communicate with you in a language you understand and do stuff that you command them to. Often, on their own.
AGI will have its own telos. An LLM does not (at least I refuse to believe that it does).
So, yeah, you could think of LLMs as programs processing data. But there is definitely some emergent behavior you notice that emerges when the LLM runs. How you interpret all this depends on your view on consciousness and intelligence.
I also don't really care whether they're conscious, but I do think it's worth challenging the claim that they are.
LLMs are programs processing data. Lots of programs exhibit emergent behavior. A simple example is Conway's game of life. It can be programmed in less than an hour, exhibits lots of interesting behavior. In fact you could run LLMs inside of it if you wanted to.
I'm not denying that LLMs are unbelievably cool and useful. It still blows my mind when I send a half-assed complex screenshot and a vague description and it just gets it and fixes the issue in seconds. It's like magic. But it isn't magic, it's just some relatively simple algorithms processing a crapload of data, and I think it's useful to understand and remember that.