I think Searle's Chinese Room argument refutes this. LLMs are simply manipulating symbols, they do not have semantic understanding. This is why hallucinations exist. And Searle's argument extends even further than LLMs.
You are basically arguing for a functional account of consciousness, but things like this have been debated for literally decades/centuries in philosophy.
Millenia, in fact. The big difference, of course, being that we now have experimental philosophy machines (aka computers). So we can actually put some of these theories to the test, and recognize how utterly inadequate most of the work done on the subject has been. We had a pretty good idea anyway, so it's not a big surprise. Theories of mind have evolved dramatically in the late 20th century. And it's pretty clear that theories of mind will have to be re-done all over again with the advent of LLMs (particularly current-generation LLMs).
The problem with the hallucination argument is (1) that is much less of a problem with good current generation AIs, and (2) living conscious breathing human beings also have a disturbing tendency to make shit up, too. So a tendency to make stuff up doesn't really serve as a disqualifier for consciousness.
Also worth mentioning that the guiding rule of what's philosophical or not is whether it's actually useful. Actually useful philosophy usually becomes something else. Usual some scientific discipline or another. And as it turns out, theories of mind are likely to become extremely useful in the near future. Expect huge advances!
I think one could argue the opposite.
1) Good current generation AIs are specifically trained to reduce hallucinations. If we had new AI system that happened to not have hallucinations as a side effect of their training, then it would be convincing. But here, it looks like we have built a pocket calculator that answer 7+13 = 14, and on top of it, we added a layer that says "if the input is 7+13, then replace the output by 20". This pocket calculator still does not know how to calculate, we just added a layer to hide its mistakes.
2) Not only "make shit up" is not the same as "hallucination" (either "making shit it" is done when the individual knows it is unreliable, or when the individual was given wrong inputs), but the point is not to say "hallucination implies no consciousness", but "large quantities of hallucinations in situations where a conscious system would be unlikely to hallucinate implies no consciousness"
>LLMs are simply manipulating symbols, they do not have semantic understanding.
falsify this. Show me a way you'd be able to prove they do/don't, that would work for humans.
Searle's Chinese Room argument is wrong.
This is not helpful. In what way is it wrong? Does the person in the room know Chinese?
It is a helpful pointer for people who might otherwise assume that a well-known argument by a famous philosopher is sound without checking too deeply. Straightforward refutations can be found on wikipedia or by thinking about it.
That just isn't true, there are no straightforward refutations of the Chinese Room that are widely accepted. Philosophers disagree about it. It's highly controversial and pretending that it's decided one way or another is not a helpful pointer for anyone.
>That just isn't true, there are no straightforward refutations of the Chinese Room that are widely accepted.
Yes there is, the systems reply is the obvious and correct answer. Philosophers that disagree are simply wrong. In the end what matters is what's true or false, not how many philosophers accept something. You can check for yourself by reading the argument, following its reasoning, and seeing that it is false; and reading the systems reply, following its reasoning, and seeing that it's true (https://plato.stanford.edu/entries/chinese-room/#SystRepl). The case is similar to those mathematical or logical proofs for the existence of god, where obviously fallacious reasoning gets a pass because it confirms deeply held beliefs.
edit: by the way as to your assertion that the argument is controversial and there is no consensus, I just found something funny on wikipedia (https://en.wikipedia.org/wiki/Chinese_room#History):
>Most of the discussion consists of attempts to refute it. "The overwhelming majority", notes Behavioral and Brain Sciences editor Stevan Harnad,[f] "still think that the Chinese Room Argument is dead wrong".[13] The sheer volume of the literature that has grown up around it inspired Pat Hayes to comment that the field of cognitive science ought to be redefined as "the ongoing research program of showing Searle's Chinese Room Argument to be false".[14]
What you are referring to is Searle assertion that "because the Chinese room concept, I conclude that every future human-made systems will be a Chinese room and will never be 'intelligent'".
I think it is an important nuance.
You have to be careful when saying "Searle Chinese room" is dead wrong: the Chinese room concept in itself is useful and not controversial, and it is possible that current LLM are "Chinese rooms", and therefore not 'intelligent'.
We could use the "Chinese room" term to denote a system that superficially mimicks human speech, but breaks down at some point and/or uses different mechanisms such that it doesn't result in consciousness. But I don't think that was the intent of the argument and it's not how the argument is generally understood in the literature, so it would just be confusing IMO.
(And you still seem to be implicitly accepting that the basic argument is valid, which would be wrong.)
> You can check for yourself by reading the argument, following its reasoning, and seeing that it is false; and reading the systems reply, following its reasoning, and seeing that it's true
You are being tedious. I obviously have done this and I disagree with you. Saying that X is logically true and Y is logically false is not a demonstration of those baseless assertions. This is not helpful, what you're saying isn't true, and what I'm saying is backed up by the wikipedia article. The bit you quote is simply stating that most literature about the Chinese Room is an attempt to refute it, which is obvious, because the people who are convinced see no need to publish saying so. The fact that people keep publishing means that they have not yet succeeded in refuting it.
Or I can simply say this: you've made a mistake in your logic. Actually, the Chinese Room argument is correct. Since you won't explicate your logic, neither will I.
Have a good day.
Ok, I think it's clear where we both stand, a good day to you too :)