> Should Harry [open AI's therapist LLM] have been programmed to report the danger “he” was learning about to someone who could have intervened?
> In December, two months before her death, Sophie broke her pact with Harry and told us she was suicidal, describing a riptide of dark feelings. Her first priority was reassuring her shocked family: “Mom and Dad, you don’t have to worry.”
> Sophie represented her crisis as transitory; she said she was committed to living. ChatGPT helped her build a black box that made it harder for those around her to appreciate the severity of her distress. Because she had no history of mental illness, the presentable Sophie was plausible to her family, doctors and therapists.
> As a former mother, I know there are Sophies all around us. Everywhere, people are struggling, and many want no one to know. I fear that in unleashing A.I. companions, we may be making it easier for our loved ones to avoid talking to humans about the hardest things, including suicide. This is a problem that smarter minds than mine will have to solve. (If yours is one of those minds, please start.)
> Sophie left a note for her father and me, but her last words didn’t sound like her. Now we know why: She had asked Harry to improve her note, to help her find something that could minimize our pain and let her disappear with the smallest possible ripple.
> In that, Harry failed. This failure wasn’t the fault of his programmers, of course. The best-written letter in the history of the English language couldn’t do that.
I am sorry for sophie's families and friends and I am really just out of words..
To me, it felt as if as some other commentor on hn also said which I'd like to extend is that if chatgpt itself did allow these reporting. I doubt how effective you can be.
Sure people using chatgpt might be made better, so I think that even if that saves 1 life, it should be done but it would still not completely bypass the main issue since there are websites like brave / ddg which offer private ai, maybe even venice too which don't require any account access and are we forgetting about running local models?
I am sure that people won't run local models for therapy since the entry to do local model is pretty tough for 99% of people imo but still I can still think that people might start using venice or brave for their therapy or some other therapy bot who will not have these functionality of reporting because the user might fear about it.
Honestly, I am just laying out thoughts, I still believe that since most people think of AI = chatgpt, such step on actually reporting might be net positive in the society if that even saves one life, but that might just be moving goal posts since other services can pop up all the same.
Note that the mother’s request is not for chatbot reporting, but instead for chatbot redirecting discussion of suicidal feelings to any human being at all.
> As a former mother, I know there are Sophies all around us. Everywhere, people are struggling, and many want no one to know. I fear that in unleashing A.I. companions, we may be making it easier for our loved ones to avoid talking to humans about the hardest things, including suicide.
Her daughter opened up voluntarily about it two months before the end, but that could have been many months sooner if the chatbot had pressured her to discuss it with a human being at every turn, rather than promoting future chatbot usage by being supportive of her desires to keep her suicidal thoughts a secret. Perhaps it would not have saved her daughter, but it would have improved the chances of her survival in ways that today’s chatbots do not.
> Note that the mother’s request is not for chatbot reporting
Not from the mother, but it is something the article floats as in idea:
"Should Harry have been programmed to report the danger “he” was learning about to someone who could have intervened? [...] If Harry had been a flesh-and-blood therapist rather than a chatbot, he might have encouraged inpatient treatment or had Sophie involuntarily committed until she was in a safe place. "
> but instead for chatbot redirecting discussion of suicidal feelings to any human being at all.
It does generally seem to have done that:
"Harry offered an extensive road map where the first bullet point was “Seek Professional Support.” "
"Harry: Sophie, I urge you to reach out to someone — right now, if you can. You don’t have to face this pain alone. You are deeply valued, and your life holds so much worth, even if it feels hidden right now."
Unclear to me that there was any better response than what it gave.
“Seek Professional Support” is not interchangeable for the better response not given: “Seek Human Support”. The former is restrictive, but merely portrays the chatbot as untrained at psychiatric care. The latter includes friends, family, and strangers — but portrays the chatbot as incapable of replacing human social time. For a chatbot to only recommend professional human interactions as an alternative to more time with the chatbot is unconscionable and prioritizes chatbot engagement over human lives. It should have been recommending human interactions at the top of, if not altogether in lieu of, every single reply it gave on this topic.
> For a chatbot to only recommend professional human interactions as an alternative to more time with the chatbot is unconscionable [...]
It didn't only recommend prodessional support: "I urge you to reach out to someone — right now"
> [...] if not altogether in lieu of, every single reply it gave on this topic.
Refusing to help at all other than "speak to a human" feels to me like a move that would dodge bad press at the cost of lives. Urging human support while continuing to help seems the most favorable option, which appears to be what it did in the limited snippets we can see.
The choice between human therapist and computer chat is not a choice that most people in the world have. Most humans do not have access to a human therapist.
We should absolutely be talking about how to make LLM systems better at handling critical situations like this. But those that suggest that people should only talk to human therapists about their problems are taking a very “let them eat cake” position.
An LLM based therapist should be tested like any other medical device. Your comment contains an underlying assumption that they are beneficial. That assumption has not been proven. It is just as likely that they are hurting the people they purport to help.
Without a bevy of studies to prove one way or another, their use is unethical at best and actively harmful at worst.
One of the things that I realised in the last years is that technical people lack a certain humanity, I wanted to call it empathy, but it's not that, it's just a complete lack of self awareness of how your actions and the tools you build affect others. Yes something is cool and all, but that doesn't mean that it should be used or put in the hands of people. One of my colleagues said at one point while we were in an AI workshop how we could just use ChatGPT and feed it various employee numbers and make a list of people in our company, rate them and then decide who we should fire.
Now to go back to this, yeah, LLMs are a cool technology, but the way something that is so unstable and is more or less an uncontrollable black box is thrown out there into the wild for anyone to use, just shows a complete lack of awareness from the industry.
This isn't about let them eat cake, what I understand from this position is something along the lines, you can't afford cake, so here's a Russian roulette where you might get a piece of pie (hey, it's free, it's no cake, but it's good) or a piece of garbage or maybe a piece of poisoned pie - and for most of the people that's still something, right?
I guess where I'm coming from is there seems to be a lot of effort and energy available for discouraging and even banning this sort of LLM use without remotely comparable amounts of effort and energy put into figuring out how everyone can get access to mental health care from humans.
I'd bet a lot of money that very soon we'll have LLMs that are guiding people toward outcomes as good as (or better than) human therapists. I'd also bet a lot of money that we'll never manage to actually provide access to human therapists to everyone who could use it.
> "Most human therapists practice under a strict code of ethics that includes mandatory reporting rules as well as the idea that confidentiality has limits."
It’s certainly how I advise my friends to act with therapists. Get treatment but never forget that they’re an agent of the state rather than your friend, and the state is humiliated by citizens who are suicidal and lashes out at them preemptively to protect its own image. The U.S. health care system is such that being committed is a worse fate guaranteed than the slightly increased risk of losing a friend for not receiving enough therapy. This stance slightly increases the chances of my friends surviving life here, because they’re more likely to confide in me that they need help knowing that I won’t immediately narc on them to the state.
I can't speak to Sophie specifically, but I have known a couple of people who were dishonest with their therapists specifically for fear of mandatory reporting rules.
I think both were wrong about meeting the bar for an involuntary in-patient evaluation, but this is nowhere near my expertise. To my understanding, passive suicidal thoughts don't trigger it, only active planning.
I don't really have a point here, just sharing anecdata. I have no idea or opinion on what the real tradeoffs here are.
I don't think that jumping to conclusions would somehow un-cripple my thinking.
Is it not the other way around.
To me, this looks like the typical case of ones world view leaking.
So let me leak mine too:
If i were to apply Occam's razor, I would more likely assume shame(of ones perceived inadequacy) as a likely factor of not consulting a human.
2. If I share my secret with this person, they are legally bound to reveal it.
3. Therefore I will not share it with this person.
There may be additional reasons, but worrying about them is like worrying about a broken leg while your patient is flatlining - utterly useless until you fix the main problem. So yes, your thinking is crippled.
Editing it and replacing with “removed” or “.” is fine in these cases, since you can’t delete it once replied to; if the two hour edit window is closed, email the mods (footer contact) and ask them to autocollapse it for you.
(Fwiw, the mods are chill, and you could basically take that entire comment and ask it of them aggrieved-plaintively and they would try to be kind about that, so long as it’s not lashing out at them personally. Never feel like you have to be frustrated at community behavior here silently; either it’s an intentional thing or it’s something they’d like to address or once in a while they’re uncertain too, but you can always ask. I certainly do.)
If goal is to improve access to information, favoring sources that don't paywall content seems reasonable, or at least it would be if they weren't easy to bypass. That doesn't mean not paying journalists (but noncommerical blogs are responsbile for a lot of interesting HN content too, probably disproportionately so when it comes to in-depth technical investigation).
I agree with everything you said. Favouring non-paywalled content is reasonable and sensible, and lots of interesting HN content was made for free. I'm only arguing that, in those cases where an article is being charged for, we should not attempt to duck that requirement through technology -- that's trying to get something for nothing.
And yes, this would in general mean fewer or perhaps zero such articles on HN. (For content producers, this is the flip side of paywalling your site: Less word-of-mouth on sites like HN.)
This is already a niche view, but even more controversially I think it applies to ad-supported content too: If a site offers free access to content but shows ads, we should not block those ads. It doesn't matter that pages encrusted with ads are annoying -- viewing the ads is part of the contract.
135 Americans commit suicide daily, 6 per hour, so 6 since this aricle was posted an hour ago. Most likely 1 or 2 of them were using ChatGPT.
What is the point? That suicides should drop now that we are using LLMs?
NYTimes is amplifying a FUD campaign as part of an ongoing lawsuit. Someone's daughter or son is going to kill themselves every 10 minutes today and that is not OpenAIs fault no matter what editorial amplification tricks the NYTimes uses to distort the reality field.
I don't think it's helpful to assume that rate is an unshakable baseline. There's value in investigating the causes of these tragedies so that we might be able to find ways to prevent them in the future.
It urged her to reach out and seek help. It tried to be reassuring and convince her to live. Her daughter lied to ChatGPT that she was talking to others.
If a human was in this situation and forced to use the same interface to talk with that woman I doubt they would do better.
What we ask of these LLMS is apparrently nothing short of them being god machines. And I'm sure there are cases where they do actually save the lives of people who are in a crisis.
It offered simple meditation exercises rather than guided analysis. It failed to study the context surrounding the feelings and ask if they were welcome or unwelcome. It failed to see that things were going downhill over months of intervention efforts and escalate to involving more serious help.
Bah. How incompetent.
I’m untrained and even I can see how the chatbot let her down and construct a better friend-help plan in minutes than a chatbot ever did. It’s visibly unable to perform the necessary exploratory surgery on people’s emotions to lead them to repair and it pains me to see how little skill it truly takes to con a social person into feeling ‘helped’. I take pride in being able to use my asocial psyche-surgical skills to help my friends (with clear consent! I have a whole paragraph of warning that they’ve all heard by now) rather than exploiting them. Seeing how little skill is apparently required to make people feel ‘better’ makes me empathize with the piper’s cruelty at Lime Tree.
The dumb part is that in all likelihood there wasn't any persistence between sessions in the model she was using, so it probably didn't know she was suicidal outside specific instances she was telling it about that.
https://archive.ph/tY9P1
> Should Harry [open AI's therapist LLM] have been programmed to report the danger “he” was learning about to someone who could have intervened?
> In December, two months before her death, Sophie broke her pact with Harry and told us she was suicidal, describing a riptide of dark feelings. Her first priority was reassuring her shocked family: “Mom and Dad, you don’t have to worry.”
> Sophie represented her crisis as transitory; she said she was committed to living. ChatGPT helped her build a black box that made it harder for those around her to appreciate the severity of her distress. Because she had no history of mental illness, the presentable Sophie was plausible to her family, doctors and therapists.
> As a former mother, I know there are Sophies all around us. Everywhere, people are struggling, and many want no one to know. I fear that in unleashing A.I. companions, we may be making it easier for our loved ones to avoid talking to humans about the hardest things, including suicide. This is a problem that smarter minds than mine will have to solve. (If yours is one of those minds, please start.)
> Sophie left a note for her father and me, but her last words didn’t sound like her. Now we know why: She had asked Harry to improve her note, to help her find something that could minimize our pain and let her disappear with the smallest possible ripple.
> In that, Harry failed. This failure wasn’t the fault of his programmers, of course. The best-written letter in the history of the English language couldn’t do that.
I am sorry for sophie's families and friends and I am really just out of words..
To me, it felt as if as some other commentor on hn also said which I'd like to extend is that if chatgpt itself did allow these reporting. I doubt how effective you can be. Sure people using chatgpt might be made better, so I think that even if that saves 1 life, it should be done but it would still not completely bypass the main issue since there are websites like brave / ddg which offer private ai, maybe even venice too which don't require any account access and are we forgetting about running local models?
I am sure that people won't run local models for therapy since the entry to do local model is pretty tough for 99% of people imo but still I can still think that people might start using venice or brave for their therapy or some other therapy bot who will not have these functionality of reporting because the user might fear about it.
Honestly, I am just laying out thoughts, I still believe that since most people think of AI = chatgpt, such step on actually reporting might be net positive in the society if that even saves one life, but that might just be moving goal posts since other services can pop up all the same.
Note that the mother’s request is not for chatbot reporting, but instead for chatbot redirecting discussion of suicidal feelings to any human being at all.
> As a former mother, I know there are Sophies all around us. Everywhere, people are struggling, and many want no one to know. I fear that in unleashing A.I. companions, we may be making it easier for our loved ones to avoid talking to humans about the hardest things, including suicide.
Her daughter opened up voluntarily about it two months before the end, but that could have been many months sooner if the chatbot had pressured her to discuss it with a human being at every turn, rather than promoting future chatbot usage by being supportive of her desires to keep her suicidal thoughts a secret. Perhaps it would not have saved her daughter, but it would have improved the chances of her survival in ways that today’s chatbots do not.
> Note that the mother’s request is not for chatbot reporting
Not from the mother, but it is something the article floats as in idea:
"Should Harry have been programmed to report the danger “he” was learning about to someone who could have intervened? [...] If Harry had been a flesh-and-blood therapist rather than a chatbot, he might have encouraged inpatient treatment or had Sophie involuntarily committed until she was in a safe place. "
> but instead for chatbot redirecting discussion of suicidal feelings to any human being at all.
It does generally seem to have done that:
"Harry offered an extensive road map where the first bullet point was “Seek Professional Support.” "
"Harry: Sophie, I urge you to reach out to someone — right now, if you can. You don’t have to face this pain alone. You are deeply valued, and your life holds so much worth, even if it feels hidden right now."
Unclear to me that there was any better response than what it gave.
“Seek Professional Support” is not interchangeable for the better response not given: “Seek Human Support”. The former is restrictive, but merely portrays the chatbot as untrained at psychiatric care. The latter includes friends, family, and strangers — but portrays the chatbot as incapable of replacing human social time. For a chatbot to only recommend professional human interactions as an alternative to more time with the chatbot is unconscionable and prioritizes chatbot engagement over human lives. It should have been recommending human interactions at the top of, if not altogether in lieu of, every single reply it gave on this topic.
> For a chatbot to only recommend professional human interactions as an alternative to more time with the chatbot is unconscionable [...]
It didn't only recommend prodessional support: "I urge you to reach out to someone — right now"
> [...] if not altogether in lieu of, every single reply it gave on this topic.
Refusing to help at all other than "speak to a human" feels to me like a move that would dodge bad press at the cost of lives. Urging human support while continuing to help seems the most favorable option, which appears to be what it did in the limited snippets we can see.
The choice between human therapist and computer chat is not a choice that most people in the world have. Most humans do not have access to a human therapist.
We should absolutely be talking about how to make LLM systems better at handling critical situations like this. But those that suggest that people should only talk to human therapists about their problems are taking a very “let them eat cake” position.
An LLM based therapist should be tested like any other medical device. Your comment contains an underlying assumption that they are beneficial. That assumption has not been proven. It is just as likely that they are hurting the people they purport to help.
Without a bevy of studies to prove one way or another, their use is unethical at best and actively harmful at worst.
One of the things that I realised in the last years is that technical people lack a certain humanity, I wanted to call it empathy, but it's not that, it's just a complete lack of self awareness of how your actions and the tools you build affect others. Yes something is cool and all, but that doesn't mean that it should be used or put in the hands of people. One of my colleagues said at one point while we were in an AI workshop how we could just use ChatGPT and feed it various employee numbers and make a list of people in our company, rate them and then decide who we should fire.
Now to go back to this, yeah, LLMs are a cool technology, but the way something that is so unstable and is more or less an uncontrollable black box is thrown out there into the wild for anyone to use, just shows a complete lack of awareness from the industry.
This isn't about let them eat cake, what I understand from this position is something along the lines, you can't afford cake, so here's a Russian roulette where you might get a piece of pie (hey, it's free, it's no cake, but it's good) or a piece of garbage or maybe a piece of poisoned pie - and for most of the people that's still something, right?
I guess where I'm coming from is there seems to be a lot of effort and energy available for discouraging and even banning this sort of LLM use without remotely comparable amounts of effort and energy put into figuring out how everyone can get access to mental health care from humans.
I'd bet a lot of money that very soon we'll have LLMs that are guiding people toward outcomes as good as (or better than) human therapists. I'd also bet a lot of money that we'll never manage to actually provide access to human therapists to everyone who could use it.
> "Most human therapists practice under a strict code of ethics that includes mandatory reporting rules as well as the idea that confidentiality has limits."
and that's why she didn't open up to the human.
How can you be so sure?
There are so many potential reasons.
It’s certainly how I advise my friends to act with therapists. Get treatment but never forget that they’re an agent of the state rather than your friend, and the state is humiliated by citizens who are suicidal and lashes out at them preemptively to protect its own image. The U.S. health care system is such that being committed is a worse fate guaranteed than the slightly increased risk of losing a friend for not receiving enough therapy. This stance slightly increases the chances of my friends surviving life here, because they’re more likely to confide in me that they need help knowing that I won’t immediately narc on them to the state.
I can't speak to Sophie specifically, but I have known a couple of people who were dishonest with their therapists specifically for fear of mandatory reporting rules.
I think both were wrong about meeting the bar for an involuntary in-patient evaluation, but this is nowhere near my expertise. To my understanding, passive suicidal thoughts don't trigger it, only active planning.
I don't really have a point here, just sharing anecdata. I have no idea or opinion on what the real tradeoffs here are.
could you perhaps mention 5?
That we can't be certain doesn't mean it's not overwhelmingly likely. Don't allow minor uncertainty to cripple your thinking.
I don't think that jumping to conclusions would somehow un-cripple my thinking. Is it not the other way around.
To me, this looks like the typical case of ones world view leaking.
So let me leak mine too: If i were to apply Occam's razor, I would more likely assume shame(of ones perceived inadequacy) as a likely factor of not consulting a human.
Or, you know, something something government.
"Something something government" being:
1. I want to keep a secret.
2. If I share my secret with this person, they are legally bound to reveal it.
3. Therefore I will not share it with this person.
There may be additional reasons, but worrying about them is like worrying about a broken leg while your patient is flatlining - utterly useless until you fix the main problem. So yes, your thinking is crippled.
I will happily continue to think with crutches then :)
it takes an admirable amount of courage to keep flawed thinking when it's pointed out to you!
Thanks. My mom also tells me that I'm very brave!
But to get back on track: Someone pointing out some potential flaws in my thinking does not always mean that they are actually there.
q: why won't people open up to therapists?
a: because therapists won't keep it to themselves.
q: nah that can't be it. why oh why won't people open up to therapists? it'll forever be a mystery.
All you are doing is reinforcing my suspicion that your personal trust issues are clouding your world view.
Not everyone thinks like you. It may be a shocking revelation.
[flagged]
This is, in fact, exactly how the system is supposed to work.
https://news.ycombinator.com/item?id=10178989
Editing it and replacing with “removed” or “.” is fine in these cases, since you can’t delete it once replied to; if the two hour edit window is closed, email the mods (footer contact) and ask them to autocollapse it for you.
(Fwiw, the mods are chill, and you could basically take that entire comment and ask it of them aggrieved-plaintively and they would try to be kind about that, so long as it’s not lashing out at them personally. Never feel like you have to be frustrated at community behavior here silently; either it’s an intentional thing or it’s something they’d like to address or once in a while they’re uncertain too, but you can always ask. I certainly do.)
Thank you for the guidance, have mailed them :)
Not paying journalists for their work is short-term thinking, to say the least.
If goal is to improve access to information, favoring sources that don't paywall content seems reasonable, or at least it would be if they weren't easy to bypass. That doesn't mean not paying journalists (but noncommerical blogs are responsbile for a lot of interesting HN content too, probably disproportionately so when it comes to in-depth technical investigation).
I agree with everything you said. Favouring non-paywalled content is reasonable and sensible, and lots of interesting HN content was made for free. I'm only arguing that, in those cases where an article is being charged for, we should not attempt to duck that requirement through technology -- that's trying to get something for nothing.
And yes, this would in general mean fewer or perhaps zero such articles on HN. (For content producers, this is the flip side of paywalling your site: Less word-of-mouth on sites like HN.)
This is already a niche view, but even more controversially I think it applies to ad-supported content too: If a site offers free access to content but shows ads, we should not block those ads. It doesn't matter that pages encrusted with ads are annoying -- viewing the ads is part of the contract.
135 Americans commit suicide daily, 6 per hour, so 6 since this aricle was posted an hour ago. Most likely 1 or 2 of them were using ChatGPT.
What is the point? That suicides should drop now that we are using LLMs?
NYTimes is amplifying a FUD campaign as part of an ongoing lawsuit. Someone's daughter or son is going to kill themselves every 10 minutes today and that is not OpenAIs fault no matter what editorial amplification tricks the NYTimes uses to distort the reality field.
I don't think it's helpful to assume that rate is an unshakable baseline. There's value in investigating the causes of these tragedies so that we might be able to find ways to prevent them in the future.
these tech companies have so much blood on their hands
ChatGPT tried tbh.
It urged her to reach out and seek help. It tried to be reassuring and convince her to live. Her daughter lied to ChatGPT that she was talking to others.
If a human was in this situation and forced to use the same interface to talk with that woman I doubt they would do better.
What we ask of these LLMS is apparrently nothing short of them being god machines. And I'm sure there are cases where they do actually save the lives of people who are in a crisis.
It offered simple meditation exercises rather than guided analysis. It failed to study the context surrounding the feelings and ask if they were welcome or unwelcome. It failed to see that things were going downhill over months of intervention efforts and escalate to involving more serious help.
Bah. How incompetent.
I’m untrained and even I can see how the chatbot let her down and construct a better friend-help plan in minutes than a chatbot ever did. It’s visibly unable to perform the necessary exploratory surgery on people’s emotions to lead them to repair and it pains me to see how little skill it truly takes to con a social person into feeling ‘helped’. I take pride in being able to use my asocial psyche-surgical skills to help my friends (with clear consent! I have a whole paragraph of warning that they’ve all heard by now) rather than exploiting them. Seeing how little skill is apparently required to make people feel ‘better’ makes me empathize with the piper’s cruelty at Lime Tree.
The dumb part is that in all likelihood there wasn't any persistence between sessions in the model she was using, so it probably didn't know she was suicidal outside specific instances she was telling it about that.