Not everyone is already under the care and watch of a professional, as I’m sure you’re aware. This can be for many reasons.
Many people are now turning to AI to vent and for advice that may be better suited for a professional. The AI is always available and it’s free. Two things the professionals are not.
From this point of view, you need to meet people where they are. When someone searches in Google for suicide related stuff, the number for the suicide hotline comes up. Doing something similar in AI would make sense. Maybe not have AI try and walk someone through a crisis, but at the very least, direct them to people who can help. AI assisting in the planning of a suicide is probably never a good path to go down.
If you can at least agree with this, then maybe you can partner with people in tech to produce guardrails that can help people, instead of berating someone for sharing an idea in good faith to try and help avoid AI leading to more suicides.
Thank you for that reply this is exactly the kind of thing I want I know people who have attempted suicide because they couldn't access help because they didn't know who to talk too or where to go, AI is in a position to provide this info they can search through 100's or 1000's of websites in the time we search one. This is exactly what i'm suggesting using the steps as distraction/take there mind of it for just a second to seek help, and like this person says i'm suggesting in good faith.