This is "AGI's finest".
It's what we all wanted. Replacing your human support team to be run exclusively by AI LLM bots whilst they hallucinate to their users. All unchecked.
Now this bug has now turned into a multi-million dollar mistake and costed Cursor to lose millions of dollars overnight.
What if this was a critical control system in a hospital or energy company and their AI support team (with zero humans) hallucinated a wrong meter reading and overcharged their customers? Or the AI support team hallucinated the wrong medication to a patient?
Is this the AGI future we all want?
Are we sure that the damage is in “millions”? Could it not be just the same old “vocal and loud minority”, who will wake up tomorrow and re-activate their subscription before the lunch break once Cursor team comes and writes a heartfelt apology post (well generated by AI with a system prompt of “you are a world class PR campaign manager …”)?
You're right, but this is more general - it has nothing to do with AGI and everything to do with poor management. It reminds me very much of the Chernobyl disaster, and the myriad examples in Taleb's "The Black Swan".
Where are you seeing AGI?
It's a Reddit post with 65 upvotes. Where are people getting millions of dollars overnight? HN is too dramatic lol
Even if they generously compensate locked out users, they'll probably end up ahead using the AI bot.