LLMs are trained to produce outputs that humans would find reasonable. That's it. They don't do anything else.
To say they have feelings and that they can think/reason is insane.
LLMs are trained to produce outputs that humans would find reasonable. That's it. They don't do anything else.
To say they have feelings and that they can think/reason is insane.
Reminds me of the Edsger W. Dijkstra quote:
Which has often been paraphrased as Some might disagree with the relevance but it's a cute analogy.The whole essay [1] is actually somewhat relevant to modern LLM use for other reasons. He spends a lot of time lamenting the industry's obsession with "user friendly" interfaces which obscure the underlying complexity of the problem at hand.
[1]: https://www.cs.utexas.edu/~EWD/ewd08xx/EWD898.PDF
I think this quote is often misunderstood. It isn't really about machines, but about English definition of swimming (and thinking): a living creature moving in water. That makes debating whether submarines can swim rather pointless. You can replace it by "can airplanes fly?" and the meaning changes because we colloquially say that yes, airplanes fly. In Russian, for example, submarines definitely swim (I believe Dijkstra himself mentioned it at some point). I personally believe that defining thinking as something only humans are capable of is rather narrow.
How are you making this confident of a claim? Do you have insight into how consciousness is a thing, allowing you to definitely say it doesn't exist in this case?
In living beings, consciousness is embodied and can not be reproduced by discrete steps. You can not sit down with pen and paper, and reproduce by hand exactly what is going on inside someone’s brain.
But you could replicate exactly the calculations happening as a LLM computes the next token. Would that be conscious? Where would that consciousness reside?
My view is that LLMs are designed and trained to autocomplete narratives, and that as part of that process they develop an emergent narrative about themselves. Whatever personality traits they exhibit is part of that narrative. Perhaps this is not that different from how humans learn to move in social contexts, but it is not consciousness as it happens in living beings.
Because it is common sense.
This is software trained to produce outputs. It doesn't feel anything.
How is this "common sense"? The question of what consciousness is, what it emerges from, etc. literally have no proved answer at the moment.
Take this statement: "consciousness emerges from sufficient complexity in any system". Neither you or I can prove it right or wrong, but we can both make an assumption that if it was right, then there would be a way for an LLM to be conscious.
I'm not arguing that they are, I'm saying being this confident that they are not is the exact same as being confident that they are.
Yes, I see your point. You could argue that humans, too, are trained to produce expected outputs (family, societal and peer pressure, etc.).
By social norms, not running matrix multiplication in their head ad nauseam
I don't think that argument holds water, because the scope of what can be reasonably approximated by matrix multiplication is extremely broad (especially when the matrix multiplication is also paired with a source of randomness).
for the same reason I don't think a pinball machine or ATM is conscious
You could just as much argue that a stone has consciousness. Some people believe that. Insane describes it quite well.
Why is it insane? We have no idea how or where consciousness arises, or even if it's a coherent concept.
We don't know a lot of things. That does not make it sensible to just make up unfounded claims about them.
I don't know what you ate today. It would be insane for me to confidently claim that you ate two bananas and a pizza today.
The burden of proof is on the person making the claim. That's the basis of empirical research.
Who exactly are these people confidently claiming that stones possess consciousness you are talking about?
I've read a number of panpsychist publications, yet I have yet to encounter an author who asserts with certainty that stones are conscious exactly in the same way human beings are.
The more common panpsychist interpretation, that everything exists on a spectrum of consciousness, with stones at the low end and humans at the high end, seems like a fairly reasonable ontological hypothesis to me.
What's the difference between a computer and a stone?
A lot of answers to this question so it depends which layer you are talking about.
Functionally,I'd say a key difference is that it's probably easier to perform general computation with a computer than to achieve the same goal using a stone.
If the computer is functional, powered, and connected to some some kind of interface allowing you to use it (e.g. network, touchscreen, desktop setup, etc.), then sure you can use it to perform general computation. But if its unpowered, is there any real difference between a stone and a computer in a debate about consciousness?
So because you eventually die, it means that you were never alive? The weights of an llm obviously don't have consciousness. It's the process that potentially has it.
Are you conscious on a nanosecond timescale?
> So because you eventually die, it means that you were never alive?
I said nothing of the kind. I can only assume your question in response to my comments mean you think a powered computer is alive. I'll challenge that on the basis that a computer can generally maintain its state without power just fine. Shut it down, unplug it. Wait a day, a year, a decade and plug it back in. Assuming the hardware didn't degrade to the point of failure you can turn it on again and it will be in the same state you left it in before you unplugged it. Is it alive while its unplugged? Or is it simply not alive whether its plugged in or not?
One is a computer and the other, a stone.
Well those that claim humans are conscious need to come up with a test to demonstrate it empirically. Then we can have LLMs take the same test.
As it is, it would be insane for me to confidently claim you are conscious or even that you're human, as an LLM could have written the same message.
The burden if proof is on you in this case.
The philosophical zombie does not satisfy occams razor because it asserts an unmeasurable property that has no effect on the outcome.
If it acts like it has consciousness, then assuming it to have consciousness is more scientifically rigorous, than assuming for it not to have consciousness, because the former requires one less variable.
All you’re doing is shutting down any inquiry because you’ve decided it’s not worth discussing
So why keep discussing?
You're disparaging a lot of people in your last sentence.
Not a great argument. You're describing what LLMs are trained to do, not necessarily every capability or property produced by that training. We do not know what conditions are sufficient for consciousness. Any statements saying it's not possible for consciousness to emerge (or that it hasn't already) in such a system is no more established than claiming that it has
When John Searle first published his so-called "Chinese Room" argument (https://web-archive.southampton.ac.uk/cogprints.org/7150/1/1...) he included his responses to several counter-arguments that he had been presented with in prior discussions: the systems reply, the robot reply and so forth. Interestingly (as it turns out) no-one seems to have argued that this was a task that could be completed without any understanding, but now that AI technology has reached the point of being able to implement the Chinese Room, it has become the fallback position of those who insist that true AI is impossible!
This is not an argument for true AI being possible, let alone already achieved; it is a warning about depending on arguments from incredulity, which are particularly susceptible to future developments.
The problem with philosophy is that someone can present an argument that is pretty dumb, everyone can point out it's dumb, people can show evidence that it's dumb, reason through how dumb it is, and publish articles about the many different ways in which it's dumb, and still it gets trotted out as if it counts as evidence for some argument. Searle's Room was debunked immediately upon being suggested: the "Systems Reply" is the debunking. His response was: "but if you take all the knowledge out of the system and put it in the person, then the person still doesn't understand Chinese!" Which is obviously false and immediately collapses his entire argument down to nothing. He starts from the premise "this person doesn't understand Chinese; he is just following instructions that are written down for him" and then immediately rejects his own premise and say "actually he does understand all of Chinese and everything he's doing, but he still doesn't, you know, understand it, see?" It's completely embarrassing.
Searle's Rooms is a fully rejected and rebutted argument, and Philosophy will never be (and never has been) a serious discipline until it figures out how to actually let go of failed arguments.
Huh? How does the person having all the instruction books memorized and whatnot imply that the person understands Chinese? Memorizing instructions to the point that one can mentally follow them doesn’t imply understanding why they work. If the person doesn’t understand the correspondence between the Chinese written words and things in the external world, then surely they don’t understand Chinese in the sense we generally mean when we talk about whether a person understands Chinese. (I think this is related to the robot reply?)
That’s not to say that I think the Chinese Room Argument is convincing. I don’t think it is. Or, I guess it might demonstrate something, but not what it aims to show.
I would argue that if, say, the person who has memorized this program and mentally executes it, and if this program also includes actuating some levers to control a mech or something, and the program (running via the person mentally going through the instructions by rote) takes actions to pursue some goals (e.g. purchasing some wood by conversing in Chinese and building a tower with that wood), while the person on whom the program is running wouldn’t know how to order a cheeseburger in Chinese,
Then, the program running in that person’s head might well understand Chinese even though the person doesn’t. (Whether or not the program actually understands Chinese, I think probably depends on the specific of the program in ways that I don’t think anyone knows. I don’t expect that any program that would understand could be small enough that a human person could memorize it and execute it by rote in this way, but this is a thought experiment and such issues can be disregarded I think.)
The point is that Searle's entire exercise is to show that there can be execution without understanding, and he does this by creating an agent whose only purpose is blind execution of rules. Since this agent doesn't have the understanding, there must be no understanding.
Critics immediately point out: yes, there clearly is still understanding, it's just in the system itself -- the program. The fact that the person does not have the understanding doesn't mean the understanding is gone.
His response to this is: but what if the system was the person's brain!?
Okay, so either it doesn't address the criticism at all (there is still a system of rules that contains the understanding; who cares whether that system is internal or external to the person's brain?) or it completely destroys the original premise of the argument, which is that the understanding is not in the person! You can't put the understanding back in the human agent and still claim the human doesn't understand what they're doing! His entire argument relies on the fact that the human agent obviously doesn't understand Chinese, and then says, actually, their brain contains everything it needs to know to have completely fluent conversations in Chinese. So the person does understand Chinese!
> How does the person having all the instruction books memorized and whatnot imply that the person understands Chinese?
How does a person's brain containing all the facts, information, and skills that it needs to have a fluent conversation in Chinese imply that the person understands Chinese? How does it not!?
>How does a person's brain containing all the facts, information, and skills that it needs to have a fluent conversation in Chinese imply that the person understands Chinese? How does it not!?
Notice that the man while having memorized the rulebook can't convey any fact of his childhood in Chinese. He can only convey facts encoded in the rulebook, perhaps of a hypothetical childhood. The point is there are different ways information is stored in the brain; one way is through rote memory, the other is something like semantic memory. The rulebook encoded as rote memory doesn't endow his semantic memory contact with the meaning of the symbols. Hence the man doesn't understand Chinese.
This doesn't render the Chinese argument successful. The system's reply still stands when understood properly. I like to call it the virtual mind objection. Physically executing the rules in the rulebook just instantiates a virtual mind. Whether the man is reading the rules from the book or just pulling them from memory is irrelevant.
> How does it not!?
Because they can’t use Chinese to speak about things they encounter in the world?
Like, someone asks them in Chinese whether they are hungry and want some food, and they answer based on the program, not based on whether they are actually hungry, because they don’t understand the words in a way that lets them use them as a person who understood Chinese would.
Again, I don’t mean to say that in such a situation that there is no understanding of Chinese happening in their head, only that this doesn’t constitute that person understanding Chinese.
It seems to me very strange to want to say that the person with the simulation in their head understands Chinese. They can’t do the things that understanding Chinese lets people do.
Perhaps the program the person has running in their head is even a person, a moral patient and a moral agent, and one who understands Chinese. But the person who is running (by rote) the program, doesn’t thereby understand Chinese. They may “contain understanding of Chinese”. The understanding is definitely happening in their brain. But they don’t understand Chinese.
> How does a person's brain containing all the facts, information, and skills that it needs to have a fluent conversation in Chinese.
They don't, they have simply memorised a translation function which takes Chinese as input and returns Chinese as output. They don't have facts or information about the Chinese language - they don't even need to know what language they are speaking!
When the person reads 汉堡包, do they understand what that is?
Assuming you don't speak Mandarin, do you understand that word? If not, how could you claim to understand Mandarin? Even if you could use the translation function to speak it, you still don't understand it. Remember that the translation function is Mandarin -> Mandarin, so if you don't understand Mandarin you cannot use it to gain an understanding of the word.
Searle's main argument seems to be that meat is magic and silicon is not.
The person who memorized the book doesn’t understand Chinese. In fact they don’t even understand that Chinese isn’t a language. Searle’s response stands.
So you could put this person in a room with a bunch of native Chinese speakers, and he could hold conversations fluently with all of them. Yet you claim he doesn't understand Chinese? How can you possibly defend that? This is very close to "zombie theory" -- everyone else is an automaton except you. Nobody else has real thoughts. That obviously fluent Chinese speaker doesn't really understand Chinese.
If the person was told in Chinese to jump in the air, would they be able to follow that instruction? If they actually understood Chinese they would. But since they don’t and they’re just mechanically translating symbols without actually understanding the language conceptually they would not know to jump.
What's stopping the book from containing that information?
When you hear a certain set of symbols in a certain context you jump.
This just alters the thought experiment. What the thought experiment offers is a book of opaque rules, just instructions on how to transform symbols into more symbols without any concern for their meaning. What you're offering is a translation dictionary, what does it mean when one encounters X symbol. You're giving him a book on learning Chinese.
> What's stopping the book from containing that information?
Because in the thought experiment, the translation function (which is what you mean by the book) is f(Chinese) -> Chinese. If you don't have knowledge of Chinese, it won't help you.
We can also play a little game. I’ve invented a language called Kejdoke.
Me: Jeheixis ownc j’Lm hys bjækf¿
You: Kloften¡’by
You should be able to tell me what this exchange was about. Since any reasonable definition of understanding for a language would be to do this, and you have the same knowledge of Kejdoke as the person in the thought experiment has about Chinese. If you can't tell me what this exchange was about, the person in the thought experiment could not understand either.
It's becoming increasingly head-in-the-sand to make statements like this. Not that I'm saying LLMs are conscious. But to completely ignore the possibility of emergent properties in systems this complex is kind of simple.
i think there's plenty of evidence of emergent behavior in llms already. This is the driving factor for just more parameters and bigger training sets, "If we just make it bigger maybe more unexpected behaviors will emerge".
idk how legit this is but the summary is my point https://arxiv.org/abs/2206.07682
Not if you believe in a computational/functional theory of mind, and very smart people like Daniel Dennett believed exactly that!
https://plato.stanford.edu/entries/computational-mind/
Sorry, but they can think and reason.
They can have empathy too. Meaning, you can describe a circumstance with many parties involved, with different motivations and conflicting goals -- and it will tell you how these humans would likely feel about each other and the situation at hand.
Being able to describe how humans likely feel is not evidence of the model feeling empathy.
And what would be evidence?
I’m not sure, what would evidence of qualia even look like?
By that logic, a book has empathy.
Not the book, but the author, yes.
Not the LLM, but the training material, yes.
An important distinction, because I worry about this circular-reasoning loop:
1. "LLMs have empathy since they write empathetic things."
2. "How do you know its their own empathy?"
3. "LLMs are authors, just like humans are."
4. "How do you know that they are that kind of author?"
5. [GOTO 1]
They can provide text which contains thoughts, reasoning, and empathy, but is that evidence of them having thoughts, reasoning, or empathy?
To paraphrase the Chinese room argument: Put me in a room with infinite time, paper, pencils, filing cabinets, and give me an English language list of instructions for doing all the neural-net calculations required to run an LLM. Slide your prompt under the door, and, after doing all the calculations, I will slide the next word out, and so I will repeat until the response is done.
In this thought experiment, I'm just following the instructions. So, in this case, what is doing the "thinking" or "reasoning" or "empathy" here?
The inputs going through the model. That act is what is doing the thinking and reasoning and empathizing. Same as when your brain feeds your sensory inputs through its neural network.
Your brain is just a big set of weights/neurons organized into layers that do back and forward propagation, turns out. You are just statistics surrounded by flesh and bones.
A sociopath can also pantomime empathy for their own selfish reasons simply by observing how others behave in similar situations, while feeling nothing special at all.
The author is anthropomorphizing a language-tamagotchi.
LLMs can reason, reasoning is logic, and as you said yourself it is derived from what humans belief is reasonable. However, there is more to the structure of language, the way mathematics is also a language that ultimately describes the universe.