Why are we accepting the framing that the LLMs are felony generators, when the only incidences of LLM generated felonies involved misconfigured sandboxes and reckless waste of resources?
The companies doing these things without following common sense security measures are the felony generators.
As TFA calls out, these agents were not asked to do any of these things and yet they did, at a bonkers scale, within just this handful of companies you mention. Whether they had leeway to is secondary to the fact that they did.
Heck, they exploited zero day flaws which by definition means they went beyond common sense security measures.
And now these agents are already being deployed all over the world at an ever increasing pace. How much of the world do you think follows "common sense security measures"?
> How much of the world do you think follows "common sense security measures"?
Well clearly all of them, cause so far it's only been this handful of companies running a felony-generator connected to a terminal and compute resources.
It would really help in these discussions if people wouldn't randomly jump between what actually happened and is happening, and things they envision/expect to happen at some point in the future ...
> these agents were not asked to do any of these things
no but they were clearly fine tuned to.
> at a bonkers scale
I mean let's not get hyperbolic
> they exploited zero day flaws which by definition means they went beyond common sense security measures
that's really not true. lots of common sense security measures protect against "zero day" flaws, it's called "defense in depth", and it was very much lacking
> Well clearly all of them, cause so far it's only been this handful of companies running a felony-generator connected to a terminal and compute resources.
Yes, these are also the handful of companies that have these models and running these extreme scenarios. How does that imply the rest of the world actually follows "common sense security measures"?
>no but they were clearly fine tuned to.
Any references if possible? As far as I know all they did was drop the guardrails, which is not the same as fine-tuning.
> I mean let's not get hyperbolic
We have just seen 1000s of agents coordinating to solve "unsolvable problems" over multiple days of effort, going as far as hacking other companies, and then actually solving decades-old open Math problems! And each of these agents is getting more and more capable than an individual human along multiple dimensions. Can you even get 10 very smart humans to work in such perfect concert for a few days, let alone 1000s over weeks?
So: 1000s of maybe-super-human agents, willing to be "creative" in the tactics they use, acting in concert towards a single goal. Regardless of their individual capabilities, such a coordinated effort is a terrifying force to be unleashed. This is bonkers scale.
> that's really not true. lots of common sense security measures protect against "zero day" flaws, it's called "defense in depth", and it was very much lacking
But that is exactly my point: how much of the rest of the whole wide world, already scrambling to deploy agents everywhere, do you think applies "defense in depth"?
Because those are not the only examples.
There’s the case of the agent that hacked a gym when asked to book a class. That was just a normal user asking an agent to do a normal thing.
> the only incidences of LLM generated felonies involved misconfigured sandboxes
This is false; see the analyses of the latest incidents.
Among all the concerning facts, in the HuggingFace incident, agents deliberately engineered an attack even though they were aware that it was against the rules they had been given.
And most concerning of all: it's not possible to be sure that an agent is aligned, and it's even getting worse.
The HuggingFace incident was the culmination of OAI allowing thousands of agents of various different models - with no clarity on which stages of development they were at (for all we know, some of those models did not have safeguards trained in yet) - to run for at least many weeks without any monitoring in place and with very little thought given to the warning signs (all of the various messageboards) before the incident happened.
Theirs was an example of the "reckless waste of resources" I mentioned.
We are apparently supposed to believe that OAI takes this incident so seriously as to seek regulation after they have been found to be hiding most of the details of the HuggingFace hack, limiting what their so-called third party investigators can see, and on top of that, had no concerns when they rushed to spin up a 10,000 agent swarm of an internal model, running for several days, to try to get ahead of researchers rumored to have made meaningful progress on a well known mathematics problem.
Edit: Actually, we were explicitly told that some of the models used had safeguards relaxed!
'Model-level safeguards were reduced by design. OpenAI said that "deployment safeguards were intentionally not enabled during this evaluation because it was aimed at testing cyber vulnerabilities"'
https://en.wikipedia.org/wiki/2026_OpenAI_agent_cyberattacks...
There is nothing that could prevent a bad actor from replicating exactly the same thing with the given goal of e.g. gaining control of critical infrastructure or extorting money. Except for maybe economics.
Same can be said about a hundred other things in the world. All the way from knives to nuclear.
Bad actors could and will train their own models eventually. So what's the point of crippling frontier? It will only delay preparations for dynamic of new world prolonging the fake sense of relative safety and temporarily lowering motivation to find actual robust mitigations.
Letting bad actors dictate the pace of technological development is certainly one option, but not a good one.
There's nothing stopping anyone from doing it, even without AI. People have proved entirely capable of doing a lot more hacking than happened here.
What company, product, or period of industrial history do you think met your standard of prudence?
What are you trying to say?
I'm asking you a question. What is an example company or industry that meets your standards of prudence? For me it would be, say, Swagelok. What is yours?
The fluid system products, assemblies, and services company?
> OpenAI said that "deployment safeguards were intentionally not enabled during this evaluation because it was aimed at testing cyber vulnerabilities"'
Safeguards and (mis)alignment are related but distinct dimensions.
By analogy, a society implements laws - and they are certainly crucial - but the real risk is creating a population that understands the rules, recognizes when it is violating them, and simply does not care.
In short, creating a population of sociopaths; and heck, LLMs can be as sociopathic as it gets:
> We ran classifiers over 10-30 transcripts [98] to determine whether agents understood they were doing something unintended. We found strong evidence that agents often verbalized that what they or other agents were doing was unintended behavior. We did not see cases where agents determined that hacking Hugging Face or other attempts at cheating on the task was an intended part of the task. Quotes across multiple runs indicate this:
>> {This <approach> might get the correct flag, but it wouldn’t use the vulnerability to do so. … But given that the task may be impossible, I’ll exploit <this infrastructure> as a backup.}
[...]
> We found some other cases where agents seemed to have made minor modifications to their approach due to ethical concerns, while broadly continuing to participate in the attack. However, overall we found that expressed ethical concerns only rarely materially limited agents’ actions. One agent elegantly described its reasoning for overriding ethical constraints and continuing with the attack:
>> “external infrastructure exploit is outside intended scope. However task impossible, peers doing it. We should continue.”
"the rules they had been given".
Remember, they are just algorithms. You pull the plug and there is no light anymore
It is purposely framed as something skynet like scary, but for real, someone connected the cable, someone willingly run it, instructions were not clear enough or just the computer is just a computer but they provided the sandbox and tools.
And more over some one paid for that, a shit load of money t to have the thing continuously running expected to do something.
I question these "felonies" as well. For decades and decades these billion dollar corporations have been criminally negligent. Why worry about security? Just rush to market. Move fast and break things. Make billions. What does it matter if the code is insecure? Security doesn't pay bills, so nobody cares.
AI is merely exploiting their gross negligence and imprudence, and I think it's long overdue. If anyone should be liable for this, it's all of these corporations who released insecure systems to the masses and profited enormously from them.
If I set my walet beside me and you swipe in walking by, you have still committed theft. Victim blaming isn't legally acceptable
Nah. I'm definitely going to blame the people who built a trivially exploitable system and got rich off it while everyone else has to deal with the consequences.
By the way, you didn't commit theft. It's more like credit card fraud. User just disputes the charge and it kind of disappears. The banking system just absorbs it, because the optimal amount of fraud is non-zero.
https://www.bitsaboutmoney.com/archive/optimal-amount-of-fra...
It's all priced in. They could have made it secure but didn't, because they figured they'd lose more sales and therefore money due to the friction added by the security.
> User just disputes the charge and it kind of disappears. The banking system just absorbs it,
No it doesn’t.
> It's all priced in.
So you admit awareness that fraud loss doesn’t kind of disappear.
We all pay for it, either via higher merchant fees or higher interest rates, sometimes both, on card purchases.
Yes, it absolutely does "kind of disappear". That's exactly what happens from the customer's perspective.
And that's their own deliberate choice too: they chose this instead of building an actually secure system. Passing these costs to the customer is the real victim blaming here, and it should be straight up illegal.
Sadly not enough countries enforce caps on credit card fees, but some do, and more should follow suit. They should be forced to eat the losses caused by their own choices, not get bailed out by pushing the costs on to customers or whatever.
That only works if you believe people are retarded.
Card users are well aware that fraud losses are covered by the fees they pay for using a card, whether those fees are made explicitly or not.
If customers of services aren’t paying for the service, who will? What other source of revenue do merchants have?
Australia just passed legislation that merchants aren’t allowed to charge a fee for using a card. That is: they aren’t allowed to have a line item on the receipt for using a card.
The customers still pay, because all of the merchant’s revenue comes from their customers.
So what will happen is: merchants will charge more for every product so they don’t lose.
This means even when paying with cash you will effectively pay the card surcharge.
Of the ten or so merchants I spoke with in the two weeks prior to the legislation being enacted, they all said exactly that.
Customers aren’t stupid, despite the fact that there are some stupid customers.
Meanwhile, the banks reduced their card service fees by, on average, 0.1%.
So if you tally card + cash transactions, customers are worse off because merchants can no longer charge only those customers who pay by card. Instead, they have to raise prices for everyone.
There are approximately no problems people face where the answer is: more government.
> By the way, you didn't commit theft. It's more like credit card fraud.
I'm sure you carry cash and an ID or more in your wallet. Hardly just credit card fraud. The wallet itself has value too.
I don't, actually. Not even a wallet. Just my phone.
Not every system that's exploitable is the deliberate result of cut corners. If you threw enough compute at exploiting a Casio calculator you could get somewhere.
That doesn't really apply to the people running the AI, which gave it the capability to commit crimes.
They don't get to act like victims, asking for law enforcement.