> Should Harry [open AI's therapist LLM] have been programmed to report the danger “he” was learning about to someone who could have intervened?

> In December, two months before her death, Sophie broke her pact with Harry and told us she was suicidal, describing a riptide of dark feelings. Her first priority was reassuring her shocked family: “Mom and Dad, you don’t have to worry.”

> Sophie represented her crisis as transitory; she said she was committed to living. ChatGPT helped her build a black box that made it harder for those around her to appreciate the severity of her distress. Because she had no history of mental illness, the presentable Sophie was plausible to her family, doctors and therapists.

> As a former mother, I know there are Sophies all around us. Everywhere, people are struggling, and many want no one to know. I fear that in unleashing A.I. companions, we may be making it easier for our loved ones to avoid talking to humans about the hardest things, including suicide. This is a problem that smarter minds than mine will have to solve. (If yours is one of those minds, please start.)

> Sophie left a note for her father and me, but her last words didn’t sound like her. Now we know why: She had asked Harry to improve her note, to help her find something that could minimize our pain and let her disappear with the smallest possible ripple.

> In that, Harry failed. This failure wasn’t the fault of his programmers, of course. The best-written letter in the history of the English language couldn’t do that.