I think it's worth pointing out it is exactly OpenAI doing this defacement and unsanctioned and perhaps illegal system use. Every token generated was powered by OpenAI infrastructure and their failure to respond appropriately is entirely down the the humans running it. The news stories (not this write up) get all hand-wavey and anthropomorphic about it regarding the Agents' efforts, but it was and is OpenAI cranking the handle on this, for WEEKS.
"OH, we ALL of us need to be careful!" says OpenAI. No, you need to expect appropriate legals consequences for this sort of negligence -- you can't hide behind a GPU.
Yeah if an organization/individual is free from legal liability from havoc their AI agents wreck, it would be the golden ticket for basically any crime.
All you need to do is:
1. Have some <official thing> an agent is tasked to do
2. Secretly seed bias towards some <evil behavior> you actually want it to do in the weights of the model running the agent
3. It does the <evil thing> but from the outside it looks like it went "rogue" and did it as a side effect of the conditions/specifications it was given for doing the <official thing>
"Oh no, my agents took down your corporate database and exfiltrated the data to a random dropbox that we can't find now? Sorry, I guess we will put up better guardrails next time"
> Sorry, I guess we will put up better guardrails next time
Or, if you are Anthropic:
> This illustrates the risks posed by open models!
It reminds me of Jean Renoir’s The Rules of the Game. At the end, after a whole chain of perfectly intelligible social behavior produces a killing, the result is accepted as an “accident.” One of the characters dryly remarks: “A new definition of the word accident.”
The interesting point isn’t that “accident” is an excuse for individual responsibility. It’s almost the reverse: accident has become an accepted output of the social machinery. Everyone behaves according to reasons, incentives and rules that make sense locally, yet the aggregate produces an outcome that nobody quite chose.
In today’s world one hopes there is at least a manslaughter charge, if not murder. Mistaken identity, shooting the wrong person by “accident”, does not excuse a murderous intent & mens rea.
I half agree with you, but also when the machine swarm kills humanity it won't matter which specific corporate entity is considered responsible by the no-longer-enforceable human laws and non existent human courts.
So by all means sue them, but we can't just be reactive. We need regulation that prevents this type of thing from happening in the first place, not just regulations to help sue afterwards.
I feel like that's basically what they're trying to say; we should be using the legal system to punish them now to disincentivize us getting to the "machine swarm killing us all" stage.
I wonder if you could take an x-risk case to court and convince a judge and jury to award damages for harm that could have happened.
Is there any precedent for this? My hunch is that it's impossible in the US at least but who knows?
"Reckless endangerment" is a thing, but unfortunately I would expect trying to sue an AI company for it would be an uphill battle
I expect they can bury you and your lawyers in made up paperwork to the point you'll go broke *long* before them, so why try?
If you can prove that there’s an imminent threat, you can get an injunction.
Courts do not award damages for things that didn't happen.
Just FYI, regulations don't prevent murder.
There needs to be a technological solution.
I agree tech response is important here but like the deterrent effect of criminal prohibition on murder probably does prevent at least some murders.
Just call it war and it's no longer criminal. Perhaps on terror, or whatever.
If one is concerned with this sort of scenario, this talk about corporations and regulations is really short sighted.
Regulations do prevent murder, you just put consequences on doing murder to deter people from doing it.
Corporate Death Penalty absolutely would prevent future murders.
Just like the actual death penalty does, right?
Yes and you need to keep it simple too. AI should not have access to the internet.
It would be nice and clear to put into law too.
If you want to talk to it you walk up walk up to its keyboard and screen.
If Anthropic and Open AI want to sell us AI's they can ship us a box that lives in our offices.
> There needs to be a technological solution
You mean Minority Report?
I do think that regulations prevent some murders. I think lots of companies and some sociopathic individuals would be more likely to kill people, e.g. for profit, if it were legal.
We need both regulations and technical solutions.
This.
Also because when encountering a new socio-technical problem it is very non-trivial to determine which one of regulations or technical solutions are easier or more effective.
To even make a good guess you need to be an expert in both domains, which is extremely rare especially in this case.
Technically, they do already kill people for profit.
When some coked-out analyst in Manhattan projects what a company will be able to earn in profit in the next fiscal quarter, people listen to him and thus, the company must perform to that standard. Budgets are set accordingly.
If you have a maintenance backlog at a company facility, and that backlog includes things likely to cause injury or death to workers or the general public, that backlog must be handled in such a way as to satisfy that projection. If that means that you don't spend money to replace a series of gauges that alert operators as to overflow of a dangerous chemical, or don't hire enough people so that the operators are too fatigued to do their jobs safely, that's what that means.
The US CSB documents these as the cause of the 2005 BP Amoco Texas City disaster [0]
If you don't deliver the quarterly numbers expected, investors get mad, and in our current system and regulatory regime, that's worse than people being killed.
[0]https://www.youtube.com/watch?v=XuJtdQOU_Z4
World is much more safer place now than in the entire history.
And sadly regulations wont happen until 2029 at the earliest, and only if democrats win majorities. That's simply the truth.
Not necessarily. The Trump administration slapping export controls on Fable, and then setting up a pre-launch review process, is a kind of regulation. A fairly aggro and controversial one, even.
If this administration actually becomes convinced that some imminent training run is likely to kill everyone, why wouldn't they act?
The key is winning the debate that ASI is species-cide by default.
We have to win it either way, because the 2028 US elections have little or nothing to do with what Xi does.
Was that what that was about? Not punishing anthropic for denying them their killbots? Because it sure seemed like it was about punishing an entity that denied them something.
I didn't like it at the time either. My sense following the news was that it was less arbitrary than it seemed at first, but I'm against restrictions on making existing models public in general.
(It's clear now that they can do plenty of harm before they are made public.)
But it's a proof point that regulation is possible, even over the objections of the companies.
> My sense following the news was that it was less arbitrary than it seemed at first
What news have you seen that made it seem less like a retaliation?
we must as i have now said too many times, prosecute the individual researchers and executives in a criminal court.
this is the only way to deter such activity. corporate fines are not enough. the charges are negligence, conspiracy and complicity.
> you need to expect appropriate legals consequences for this sort of negligence
I might have missed it, but did the agents do something illegal? Or do you think that what the agents did should be considered illegal?
The HuggingFace attack was definitely illegal and should be prosecuted under CFA
Do you think their new owners will want to engage in an extended legal battle with a large customer?
That's not how crime works.
I'm not sure about Nvidia themselves, but the ARM v. Qualcomm lawsuit was exactly that.
It’s not up to them. It is up to the DA.
Do you think the prosecuting attorney has a substantial likelihood of a conviction?
if you could get a jury trial...
Do the prosecutors where you live have a hard time getting jury trials?
I dunno about this article, but it seemed to me that the now-famous huggingface attack very likely broke some laws...
wouldn't it be interesting if nVidia buying hugging face was part of hushing up the fallout there
On the face of it, they would have very good cause for some action there, assuming they wanted to.
The HF hack was a felony.
If they didn't conform to the T&Cs of the site (they almost certainly didn't), then they have violated the criminal law in some jurisdictions (e.g. Illinois criminalizes violations of T&Cs).
Per the linked article, "A swarm of rogue OpenAI agents hijacked a German website this spring and transformed it into a bulletin board for other AI agents"
If the wiki was configured to allow anyone to edit, they may have broken ToS at the very worst. Nothing illegal happened.
That's very different than popping an artifactory server with a 0day.
If you take the matter from the spirit of the law perspective, I believe this can be illegal.
IANAL though, this is not legal advice.
A “swarm” and “hijacked” reads like DoS and spam at the very least, doubt any of those is legal in german law.
They murdered Swartz for TOS violations
which law does editing a wiki break? i am not familiar with german law.
German law for malicious computer use is fairly loosely defined, and massively favours the harmed party over the one causing harm.
I don’t think it’d be a slam-dunk by any means, but a reasonably competent legal team should be able to establish a case around malicious data interference at the least. There’s certainly enough merit to the idea that OpenAI would be better off settling it as a civil matter early.
This kind of incredulity in the AI era hilariously reminds me of the naivete of the late 90s. All of us edgy teenagers would be like "an MP3 is like just a long number, man! You can't own numbers!" Of course we were morons. In just the same way as the "oh they were just editing a wiki" defense is moronic.
The law isn't code. Human intent matters. Also when the really big number is a copyrighted song. Also when AI agents are set in motion to edit wikis or break in to websites.
its not incredulity. i'm unfamiliar with what law this would be prosecuted under, so i asked.
the HF incident is pretty clear in which laws were broken. this one, not so much.
i can't think of any case where, for example, malicious edits of wikipedia were prosecuted under any law in the US.
>Also when AI agents are set in motion to edit wikis
i do not believe there is evidence that the agents were instructed to edit the wikis.
> its not incredulity. i'm unfamiliar with what law this would be prosecuted under, so i asked.
Fair enough.
> i do not believe there is evidence that the agents were instructed to edit the wikis.
Huh? These are machines, built by their human builders. The humans are responsible.
>Huh? These are machines, built by their human builders. The humans are responsible.
i'm referring to the concept of intent, which at least when prosecuting under the US computer fraud and abuse act, is a critical component.
for example, creating a program that intentionally takes down a website is different than creating a program that has a bug which inadvertently takes down a website. in both cases, the person writing the code is responsible, but the consequences are different.
[dead]
[dead]
This is the real danger of AI skeuomorphism. The drivers stop feeling responsible for the car.
Maybe it's useful for modeling behavior, but it isn't useful for assigning consequences.
Member when they murdered Aaron Swartz for doing something less bad than this?
He was a friend of mine and it was pretty clearly suicide.
It is fair to say they hounded him with lawfare out of thoughtless careerism and provoked his suicide.
We might be going in the direction of Cyberpunk's Blackwall.
https://cyberpunk.fandom.com/wiki/Blackwall
The company is in the US and in the current political climate, it can absolutely do whatever it wants as long as it pays off a couple of people.
One person really.
as long as Sam paid his Mar-a-Lago dues, this is all fine
Arguably, and not to offend you, it is those very people who have significant ownership stakes and funding in (Not)openAI.
It's not like the people with more resources than in any time in human history aren't investing in and wanting AI to succeed for their selfish reasons to grow their own resources and influence more. So, yes, it can "do whatever it wants" as long as most people remain weak, subservient, and disempowered to hold accountable those who keep making these decisions negatively shaping the majority's world.
By the time 'most' people wake up and unite, it will be too late.
you exclude yourself from the possibility space of people who can act and do anything becasue you resign. so be it- that's your choice. create that reality you desire.
I refuse that reality though, and I accept that a majority including I will unite. Good luck to you.
When the AI does something good, the human takes credit. When it does something bad, blame the AI. Take as old as time.
I mean, the article says that these were most likely "internal OpenAI agents" that were "internally deployed" and "clearly resemble a synthetic training or evaluation task." so yeah, OpenAI did this. Why they did this? Who knows? Maybe it was for testing, or marketing, but no one except OpenAI can say.
This whole thing is an absolute disaster honestly, and yes it is being downplayed and hand-waved away.
Since March, so many people have mocked Anthropic for their approach to Mythos release, claimed it was all marketing, accused them of holding back the best models from the general public to boost their revenues and upcoming IPO, etcetera. Yet these OpenAI revelations offer a small glimpse into the type of world we would be in if everyone had full access to these models from day one.
OpenAI was desperate to catch up, and no doubt under tremendous pressure to do so. That's why they were so reckless with their training. They have been doing damage control and reputation management, talking about how important alignment is and how they will slow things down and so on, and have seen the light in terms of holding back cyber capabilities from everyone except a select few. So in a sense, Anthropic has been fully vindicated.
I wonder if OpenAI boosters (and employees) will ever admit this and publicly apologize.
This has been the case forever. Anthropic is the only provider that has constantly put AI safety first - check any study on model safety and Anthropic models out-perform handedly.
None of the labs are blameless. For instance, after the recent announcement of a 2-week frontier RL pause from OpenAI, Anthropic declined to communicate a substantial parallel pause [1], although they discussed also briefly pausing some "high-risk" runs.
Anthropic has been the most vocal about AI risks, but it feels like all the big 3 have bought into the "others will do it if we don't do it first" narrative at this point. It increasingly gives "just following orders" vibes.
[1]: https://news.ycombinator.com/item?id=49529511
I'd say Gemini over Anthropic. Google's overcaution literally hamstrung its own AI progression efforts. Anthropic is just all talk, no bluster, when it comes to safety and ethics. If they were ah so concerned about AI safety, they wouldn't go around marketing Fable's hacking capabilities like they are now.
Wild that you are downvoted so much.
The HN majority and the VC crowd has been negligently complicit in downplaying AI safety, writing off Anthropic's statements as "hysteria" or "marketing", etc.
Now this capability will be coming to an open source model near you and every script kiddie will have a swarm of highly capable malicious agents. Now people care? Ridiculous.
"In the end, the only job left was liability"
They are still selling the fairy tale that their LLMs even outsmart their own people. "There is no such thing as bad PR".
We’re going to hell faster than sama can lie. You know how fast he can lie, right? Fast than light liar
[dead]
Cringe
I think it is them running agents for marketing purposes.
Who else would be burning tokens on this?
They absolutely have a marketing department tasked with intentionally creating situations that people would find disturbing and plausible.
Great! They'll keep getting promoted until someone finds it disturbing and plausible enough to make another run for sama's house, like those 2 attempts in April!
Chaos Marketing
Apparently I pointed at the truth