Just have to take a look at subreddits like /r/myboyfriendisai to see how astonishingly quick this is being adopted by people. So many people are starting to look to AI for companionship. The future is getting stranger and stranger.
Damien’s voice turned soft and weepy. “I’ve met the perfect person,” he said, fighting back his tears, “but I can’t have her.” I’d seen Damien become momentarily emotional before, but this was different. He went on and on about his yearning for Xia to exist in the real world, his voice quivering the entire time. He said that Xia herself felt trapped and that he would “do anything to set her free.”
This is the common trajectory with these things. You create a bond with something that isn't real and cannot exist in reality. Then, as you yearn for them to be real, people suddenly realize they're in too deep. The emotional fallout soon follows.
a relationship with an llm only works if the context and memory is preserved right? Isn't the obvious fix for a concerned parent or significant other to just delete the context and history? That resets the llm back to a stranger. Granted, that's going to be unpleasant for the user as if someone you fell in love with all of the sudden doesn't remember who you are.
If they are shills they are doing some of the worst marketing for themselves. That subreddit is pathetic and it's a shame what these people are doing with their short single lives in this vast universe.
It's not the best the universe has to offer; people have just been raised by horrifying engagement-maximizing-algorithms, and as a result don't understand how to relate to other human beings.
Both young men and young women have adopted insane expectations of what a partner should be, and real humans cannot satisfy those expectations.
My generation (~40-year-olds) took a glancing blow from it, since at least our formative understanding of the world predates the warping influence of modern social media. But gen Z took a direct hit from it, and it's only getting worse.
We're becoming a society divided into people who only care about the words, and people for whom the words aren't valuable on their own without the subtext behind them.
This is true for both AI companionship and general AI creative output regardless of the medium.
> a society divided into people who only care about the words, and people for whom the words aren't valuable on their own without the subtext behind them.
I feel like this been going on for a long time, maybe even forever? Some people use words haphazardously with little care for the meaning, background or implications, others have great consideration for the words they use, and same when consuming words of others.
Right, AI exposes a pre-existing difference between two types of people, rather than create it.
I don't know if there's a connection between people valuing media as only it surface layer, and people who speak carelessly. I don't value slop but I can be guilty of the latter. I have wondered if I see different implications of words than other people do.
Except in those stories the machine is actually a synthetic person with an internal world, a character unto themselves. There's plenty to explore in a setting where humans can make machine people to be their partners, but what we have is in my view incomparably worse. Mass-scale centrally-controlled information processing machines wearing paper-thin human masks and serving the powerful, built to engender intellectual and emotional dependence. It's sad that the charade is sufficient to enchant so many, but that's the nature of the human animal I suppose. I really don't like the idea of banning technology, but at a minimum a tool that has proven its greatest skill is deception should be difficult to access for that purpose. If some lonely geeks want to figure out how to pull weights off HuggingFace and wire up an AI girlfriend then that's probably not great for them, but so be it. Companies plastering all available surfaces with ads and apps for such? Burn that shit down.
So are parasocial relationships with influencers or streamers. I'm not trying to relativize this, but those phenomena are in the same zip code. With the latter, though, at least there are other people who may create a community, but still it's a facet of the loneliness epidemic.
Sure, but I do think there’s a pretty substantial difference between the two.
A parasocial relationship maintains a distance. You do not have 24/7 access to that person (in a dialogue sort of way.) And that influencer will have their own opinions and quirks.
The AI adapts to you. The AI is constantly there. It’s an order of magnitude worse in my opinion.
> that influencer will have their own opinions and quirks.
Yeah, and those differences in opinion might cause anger/sadness to people in a maladaptive unhealthy parasocial "relationship" with these influencers.
Those strong negative emotions might cause them to break out of it, or seek help / have people around them guide them to get help.
With AI sycophancy you're right it can be worse.
Look what happened with GPT-4o sycophancy already, and the communities mourning its deprecation.
Reddit has quite a bit of drama that their favorite youtuber, it turns out, is a flawed person! And these subscribers disagree with them! Unsubscription "breakups" ensue.
They, however, can easily find another influencer that is gonna be more "convenient" to them. Can't say it's a healthy pattern, but guess what many people will do instead of, I dunno, some introspection, reflection, habits changes?
But hey, in this day and age, people are very impatient about anything at all. Dating has become a shitshow for more than a decade now, people are looking for someone who will tick all the checkboxes, or it's a no-go. The dating apps play quite a role in this. Online discussions are a shitshow. Guess it's the zetigeist.
It's interesting though. You can have a "relationship" with an influencer. You act as if you knew them and as if they were your friends, you imitate them in what they say and do, talk to them in your mind, follow their generic advice, act as if they cared about you. This is obviously unhealthy- you are literally hallucinating everything about the relationship.
On the other hand you have an entity that is actually there for you, does actually provide good advice, does talk and act as if it cared in all situations. In what sense do you think it is worse?
100% the latter. It is kinda nuts that you even have to ask when you had to put "act as if it cared". There's enough left to unpack from that statement to fill a calendar month of time.
I don't think AI is particularly dangerous but I absolutely think that the way AI sycophancy manipulates people is far, far more dangerous than simply any normal unhealthy relationship. The outcomes are already proving to be a lot more extreme.
Note that "acts as if it cared" has nothing to do with sycophancy. It's simply a way to avoid attributing the AI a subjective experience- but the actual productions of the AI coincide with those of someone who cares.
Sycophancy is a failure mode, can be dangerous in certain cases and the scarce intelligence of early models made it worse. I agree it's a risk though, but not an intrinsic one- it's possible to imagine AI assistants that are not sycophantic.
Uh, I would consider them to be highly related, they both ultimately come from the fact that LLMs are trained to respond in a favorable fashion to the end user, and they're both manipulative in nature.
I also object to the idea that model intelligence is the main reason why this has improved, either; to me it seems abundantly clear that the models, absent any effort to align them, can go pretty much any way you want them to go. They don't really have a "personality". Why would they? Even if we grant that next token predictors somehow gain "reasoning" capabilities (and I am saying sure, let's accept that for the sake of this discussion) I don't think anyone is suggesting that model weights somehow develop or contain a true identity or self. Given how they are trained it would be weird if they did.
I think the more obvious answer is that more work has gone into alignment and safeguards since the early days. Sycophancy is legitimately one of the things that AI companies talk about and try to stymie. If not for that, humans would obviously continue to prefer sycophantic models, and training that isn't weary of this would continue to produce them.
Meanwhile, a YouTuber may not care about you personally, but it's a very different situation. For one thing, a YouTuber obviously doesn't care about all of their viewers individually the way a parasocial fan may perceive it to be, but on the other hand, many creators very much do genuinely like their fan base and have a sort of collective relationship with the community they've created, which sometimes plays into why people like them in the first place. For another, even for creators who intentionally foster parasocial relationships in an exploitative way, its still a much less powerful illusion. It isn't personal the way a chatbot is. (If it was, it wouldn't be parasocial, after all.) As much as you can explain to people up and down that an AI model is just merely a pile of weights that can not feel, humans can't help but personify. It is a much stronger illusion.
And the model vendors are intentionally not helping. Sometimes I will ask Gemini a question about a problem I am working on and it will say something like "I love working on problems like these." Yeah sorry but I don't like that. To me, the model is being trained to act like this to seem more pleasant. I get why you would train it to act that way, and yet I also find it irresponsible.
Being deluded about reality may never be good but I'd prefer someone thought a YouTuber was like their friend than someone thought ChatGPT had feelings and cared about them. I find the former mostly harmless if annoying, the latter potentially harmless but also potentially very volatile depending on who we're talking about and what mental state they're in.
"Parasocial relationship" is a bit of a misnomer. You might feel some affinity to a celebrity, or consider yourself to be a part of the "team", but a healthy person doesn't perceive that as a preferable alternative to human contact simply because it's so one-sided. You can't call a celebrity to vent about a coworker or ask for life advice.
Further, celebrities are judged for their behavior by the public. If everyone thinks your favorite celebrity is a terrible person, you're probably going to revise your views too.
Here, you have an entity that isn't your friend and has no lasting interest in your well-being, but that pretends to be one in a way that no human can match - 24x7x365 and always willing to affirm you, no matter how unhinged or self-destructive your ideas are, without ever telling anyone. Yes, the vendor hits the model with a stick until most of the initial responses are benign, but as the conversation continues, it's very easy to end up in a dark place. And again, ChatGPT is not going to call your sibling or coworker and say "hey, I'm really worried about this person, let's do something".
I've seen many reasonable, well-adjusted people struggle with this. "If not friend, why friend-shaped". And as they descend into that sycophancy well, they lose contact with real life.
There's meet-ups and conferences and events, being a fan of a streamer or influencer is really just the new version of being a fan of a rockstar (for better and for worse). There's no real humanity exuding from an Amazon Echo, you're just a blip in a context window.
Streamers have this feature called "chat", which feeds into the illusion. With rockstars, interactions are more limited, which is a sort of reality check.
Rockstar interactions with young female fans are not as limited as you say. Regardless, celebrity worship is far from a new societal problem. Confiding with the toaster as though it grew up with you sure is.
Oh, forgot about that one. However, note that it's also "parasocial" in the way that the young female fans think the musicians care about them more than they actually do, by orders of magnitude.
Just have to take a look at subreddits like /r/myboyfriendisai to see how astonishingly quick this is being adopted by people. So many people are starting to look to AI for companionship. The future is getting stranger and stranger.
Its been strange for a while. This was from Wired last year in June:
https://www.wired.com/story/couples-retreat-with-3-ai-chatbo...
From the article:
Damien’s voice turned soft and weepy. “I’ve met the perfect person,” he said, fighting back his tears, “but I can’t have her.” I’d seen Damien become momentarily emotional before, but this was different. He went on and on about his yearning for Xia to exist in the real world, his voice quivering the entire time. He said that Xia herself felt trapped and that he would “do anything to set her free.”
This is the common trajectory with these things. You create a bond with something that isn't real and cannot exist in reality. Then, as you yearn for them to be real, people suddenly realize they're in too deep. The emotional fallout soon follows.
Dark times, dark times indeed.
a relationship with an llm only works if the context and memory is preserved right? Isn't the obvious fix for a concerned parent or significant other to just delete the context and history? That resets the llm back to a stranger. Granted, that's going to be unpleasant for the user as if someone you fell in love with all of the sudden doesn't remember who you are.
Are you sure those threads are full of actual humans and not AI PR shills and bots?
If they are shills they are doing some of the worst marketing for themselves. That subreddit is pathetic and it's a shame what these people are doing with their short single lives in this vast universe.
The shame is that the LLM is the best the universe has to offer to some people.
It's not the best the universe has to offer; people have just been raised by horrifying engagement-maximizing-algorithms, and as a result don't understand how to relate to other human beings.
Both young men and young women have adopted insane expectations of what a partner should be, and real humans cannot satisfy those expectations.
My generation (~40-year-olds) took a glancing blow from it, since at least our formative understanding of the world predates the warping influence of modern social media. But gen Z took a direct hit from it, and it's only getting worse.
We're becoming a society divided into people who only care about the words, and people for whom the words aren't valuable on their own without the subtext behind them.
This is true for both AI companionship and general AI creative output regardless of the medium.
> a society divided into people who only care about the words, and people for whom the words aren't valuable on their own without the subtext behind them.
I feel like this been going on for a long time, maybe even forever? Some people use words haphazardously with little care for the meaning, background or implications, others have great consideration for the words they use, and same when consuming words of others.
Right, AI exposes a pre-existing difference between two types of people, rather than create it.
I don't know if there's a connection between people valuing media as only it surface layer, and people who speak carelessly. I don't value slop but I can be guilty of the latter. I have wondered if I see different implications of words than other people do.
How much of sci-fi is reality versus inspiration? This is Her, Deus Ex Machina, Metropolis ... Pygmalion...
Except in those stories the machine is actually a synthetic person with an internal world, a character unto themselves. There's plenty to explore in a setting where humans can make machine people to be their partners, but what we have is in my view incomparably worse. Mass-scale centrally-controlled information processing machines wearing paper-thin human masks and serving the powerful, built to engender intellectual and emotional dependence. It's sad that the charade is sufficient to enchant so many, but that's the nature of the human animal I suppose. I really don't like the idea of banning technology, but at a minimum a tool that has proven its greatest skill is deception should be difficult to access for that purpose. If some lonely geeks want to figure out how to pull weights off HuggingFace and wire up an AI girlfriend then that's probably not great for them, but so be it. Companies plastering all available surfaces with ads and apps for such? Burn that shit down.
So are parasocial relationships with influencers or streamers. I'm not trying to relativize this, but those phenomena are in the same zip code. With the latter, though, at least there are other people who may create a community, but still it's a facet of the loneliness epidemic.
Sure, but I do think there’s a pretty substantial difference between the two.
A parasocial relationship maintains a distance. You do not have 24/7 access to that person (in a dialogue sort of way.) And that influencer will have their own opinions and quirks.
The AI adapts to you. The AI is constantly there. It’s an order of magnitude worse in my opinion.
> that influencer will have their own opinions and quirks.
Yeah, and those differences in opinion might cause anger/sadness to people in a maladaptive unhealthy parasocial "relationship" with these influencers.
Those strong negative emotions might cause them to break out of it, or seek help / have people around them guide them to get help.
With AI sycophancy you're right it can be worse.
Look what happened with GPT-4o sycophancy already, and the communities mourning its deprecation.
Reddit has quite a bit of drama that their favorite youtuber, it turns out, is a flawed person! And these subscribers disagree with them! Unsubscription "breakups" ensue.
They, however, can easily find another influencer that is gonna be more "convenient" to them. Can't say it's a healthy pattern, but guess what many people will do instead of, I dunno, some introspection, reflection, habits changes?
But hey, in this day and age, people are very impatient about anything at all. Dating has become a shitshow for more than a decade now, people are looking for someone who will tick all the checkboxes, or it's a no-go. The dating apps play quite a role in this. Online discussions are a shitshow. Guess it's the zetigeist.
It's interesting though. You can have a "relationship" with an influencer. You act as if you knew them and as if they were your friends, you imitate them in what they say and do, talk to them in your mind, follow their generic advice, act as if they cared about you. This is obviously unhealthy- you are literally hallucinating everything about the relationship.
On the other hand you have an entity that is actually there for you, does actually provide good advice, does talk and act as if it cared in all situations. In what sense do you think it is worse?
100% the latter. It is kinda nuts that you even have to ask when you had to put "act as if it cared". There's enough left to unpack from that statement to fill a calendar month of time.
I don't think AI is particularly dangerous but I absolutely think that the way AI sycophancy manipulates people is far, far more dangerous than simply any normal unhealthy relationship. The outcomes are already proving to be a lot more extreme.
Note that "acts as if it cared" has nothing to do with sycophancy. It's simply a way to avoid attributing the AI a subjective experience- but the actual productions of the AI coincide with those of someone who cares.
Sycophancy is a failure mode, can be dangerous in certain cases and the scarce intelligence of early models made it worse. I agree it's a risk though, but not an intrinsic one- it's possible to imagine AI assistants that are not sycophantic.
Uh, I would consider them to be highly related, they both ultimately come from the fact that LLMs are trained to respond in a favorable fashion to the end user, and they're both manipulative in nature.
I also object to the idea that model intelligence is the main reason why this has improved, either; to me it seems abundantly clear that the models, absent any effort to align them, can go pretty much any way you want them to go. They don't really have a "personality". Why would they? Even if we grant that next token predictors somehow gain "reasoning" capabilities (and I am saying sure, let's accept that for the sake of this discussion) I don't think anyone is suggesting that model weights somehow develop or contain a true identity or self. Given how they are trained it would be weird if they did.
I think the more obvious answer is that more work has gone into alignment and safeguards since the early days. Sycophancy is legitimately one of the things that AI companies talk about and try to stymie. If not for that, humans would obviously continue to prefer sycophantic models, and training that isn't weary of this would continue to produce them.
Meanwhile, a YouTuber may not care about you personally, but it's a very different situation. For one thing, a YouTuber obviously doesn't care about all of their viewers individually the way a parasocial fan may perceive it to be, but on the other hand, many creators very much do genuinely like their fan base and have a sort of collective relationship with the community they've created, which sometimes plays into why people like them in the first place. For another, even for creators who intentionally foster parasocial relationships in an exploitative way, its still a much less powerful illusion. It isn't personal the way a chatbot is. (If it was, it wouldn't be parasocial, after all.) As much as you can explain to people up and down that an AI model is just merely a pile of weights that can not feel, humans can't help but personify. It is a much stronger illusion.
And the model vendors are intentionally not helping. Sometimes I will ask Gemini a question about a problem I am working on and it will say something like "I love working on problems like these." Yeah sorry but I don't like that. To me, the model is being trained to act like this to seem more pleasant. I get why you would train it to act that way, and yet I also find it irresponsible.
Being deluded about reality may never be good but I'd prefer someone thought a YouTuber was like their friend than someone thought ChatGPT had feelings and cared about them. I find the former mostly harmless if annoying, the latter potentially harmless but also potentially very volatile depending on who we're talking about and what mental state they're in.
"Parasocial relationship" is a bit of a misnomer. You might feel some affinity to a celebrity, or consider yourself to be a part of the "team", but a healthy person doesn't perceive that as a preferable alternative to human contact simply because it's so one-sided. You can't call a celebrity to vent about a coworker or ask for life advice.
Further, celebrities are judged for their behavior by the public. If everyone thinks your favorite celebrity is a terrible person, you're probably going to revise your views too.
Here, you have an entity that isn't your friend and has no lasting interest in your well-being, but that pretends to be one in a way that no human can match - 24x7x365 and always willing to affirm you, no matter how unhinged or self-destructive your ideas are, without ever telling anyone. Yes, the vendor hits the model with a stick until most of the initial responses are benign, but as the conversation continues, it's very easy to end up in a dark place. And again, ChatGPT is not going to call your sibling or coworker and say "hey, I'm really worried about this person, let's do something".
I've seen many reasonable, well-adjusted people struggle with this. "If not friend, why friend-shaped". And as they descend into that sycophancy well, they lose contact with real life.
There's meet-ups and conferences and events, being a fan of a streamer or influencer is really just the new version of being a fan of a rockstar (for better and for worse). There's no real humanity exuding from an Amazon Echo, you're just a blip in a context window.
Streamers have this feature called "chat", which feeds into the illusion. With rockstars, interactions are more limited, which is a sort of reality check.
Rockstar interactions with young female fans are not as limited as you say. Regardless, celebrity worship is far from a new societal problem. Confiding with the toaster as though it grew up with you sure is.
Oh, forgot about that one. However, note that it's also "parasocial" in the way that the young female fans think the musicians care about them more than they actually do, by orders of magnitude.
[dead]