The issue is that people form opinions based on what LLMs spit out, they think it's "as good as google", and that google was "as good as encyclopedias" and encyclopaedias were "as good as fact"..
Totally forgetting that the accuracy goes down with each of those evolutions.
I'm also reminded of that "You are not immune to propaganda" mantra, in that, the most likely to be swayed by LLMs are those that believe that they understand that they're inaccurate and sometimes dangerous.
Case in point: myself, today, when claude accidentally ran `rm -fr $variableThatDoesntExist/*` and wiped a server of mine.
The variable was one it had set itself (or had assumed).
At some point as these LLMs get better and better we have to take a step back and remind ourselves that: despite the fact that they can be very persuasive and they can earn trust by being mostly correct: they're still capable of great harm (perhaps much greater harm) because they are dangerously wrong sometimes..
.. and worse, they are controlled by corporate interests.
agree on the general points, but what i take from this is that you gave an AI access to the whole freaking filesystem?
I don't think it's even true, at least as written. 'rm' hasn't worked that way in many years (with GNU rm you need --no-preserve-root now) and every modern harness has guardrails specifically to prevent this. if it IS true, it was some weird/hacky setup with an OSS harness on a BSD or something.
That said, I often give mine access to the full file system and YOLO mode. It's fine if you treat boxes as cattle rather than pets. I don't think a SOTA model is any more likely to blow it up than a junior engineer. Probably less so.
You're both making my point.
"Of course that's dangerous" - I'm aware of what I'm doing and the machine was entirely replaceable, but claude had been doing things with an extreme level of competence so I never considered it could do something so dangerously stupid, it had been catching much more subtle gotchas for the last 2 months building my trust.
And you truly believe that it's not even possible.
Well; https://imgur.com/CCD98fO
(Claude Code building against a Debian 13 machine over ssh using the official app on MacOS, btw.)
Let me reiterate the point: "AI can make mistakes" → "This AI seems to make very few mistakes and is catching nuances I might have missed" → "Maybe I trust this AI more than I should- but it's good!" → *AI makes major mistake*
This is as true for code as it is for internet facts.