It’s great that nuclear power safety events are so well managed that nearly 100% of events go completely unnoticed by the public.
Unironically this is the kind of headline I’m expecting in 2040 about one or more major AGI systems running an electrical grid, factory or similar.
This was a power safety event in the sense that safety was accidentally added. Like oops, we accidentally put the pin back in the grenade.
Which indicates that there’s so much understanding of how it works and the power that it uses that compensatory systems have been created to increase its reliability
what precisely is your point of describing how quality production engineering for infrastructure works?
> nearly 100% of events go completely unnoticed by the public
"Nearly" is doing a lot of heavy lifting there. Those "nearly" 0% of events which DO go noticed by the public, like you know, Fukushima, tend to be world news for weeks at a time.
I doubt AGI will be used for life-or-death systems like that if it resembles anything like the LLMs of today. Systems at that scale are about risk management, and everything we know about AI suggests that it cannot mitigate risk at all. All AI is seemingly a net-positive risk factor.
Really, the "big AI story" will be when the next Snowden defects from the NRO and tells us what SENTIENT is doing. Until then it's just going to be a bunch of pie-in-the-sky wishcasting about what we want the future to look like. It's getting a little tiring hearing people insist that abundance is around the corner while surveillance is here-and-now.
I’m absolutely willing to believe there are some people that stupid.
I doubt anyone who works on reactors is that stupid.
But some idiot somewhere is going to put some industrial stuff under AI control pretty soon and murder a few people at minimum.
The way I read it was that he expects large industrial systems with AI to have safeguards to "fail closed."
Which: it seems like we're already getting to the point of capability vs. unexpected behavior that having the ability to "trip" a complex system may be an important design criterion.
I mean, there's plenty of harm these systems can do besides trying to turn us all into paperclips.
Correct interpretation
So, which do we choose?
A) Let AI override safety mechanisms (increases uptime while endangering humans)
B) Let AI impose additional safeties (decreases uptime with questionable safety uplift)
If you intend to go with option B) then there's basically zero benefit to investing in that kind of automation. AGI will be fallible, and even with all the safeties and safeguards it's hard to see what tangible benefit AGI would add to the system. It'll be a hard sell for critical infrastructure of any kind.
I understand that it's hard to get a man to understand something when his AGI Society executive credentials depend on it, but it feels glaringly obvious why AI isn't masterminding these sorts of systems. Even accounting for unprecedented advancement in AI, it's hard to see what AI could contribute to managing these things.
> It'll be a hard sell for critical infrastructure of any kind
Remember, critical infrastructure attached SCADA systems to the internet and has had trouble removing them all despite 30 years of warnings and audits.
There's definite advantages to having a LLM in the loop. The best we can hope for is that it's architected in a way that it fails safe and damages are limited, so we get some of the advantages but less of the downside.
There are more than two options
That assumes people mistrust LLM output, which is a perspective inside the HN bubble.
Outside, almost everyone I know - including highly educated people - takes even Google's search LLM output as fact, without a second thought, and even regarding serious issues (e.g., medical problems). Nobody raises a doubt or an eyebrow. A recent article says members of US Congress use it extensively for serious work. Lawyers use it to file cases. Doctors use it to summarize essential medical information. I even hear that software developers use it to create high-risk, high-value code.
In the public people think computers are superior to humans - more authoritative, objective, perfectly logical and error-proof. SV pushes that narrative, for example saying computer-driven cars are safer because computers don't have human flaws (that old sales technique - compare only the competition's unique flaws and don't mention your own; computers have their own flaws that humans lack). And of course, that one is popular on HN.
(IMHO the way to educate the public is to spread the narrative that every computer is created (designed, programmed) by normal, flawed, biased, etc. humans, and controlled by other flawed humans for their own agenda. And that computers have their own flaws.)
The Elon buttlicker types praise llms as the new knowledge gods.
The general public distrusts them cuz they know that they're coming for their jobs. But they will use them out of convenience
The fact that you conflate LLMs and AI means you’re unqualified to be part of the discussion
[dead]
[dead]