Edits for clarity: seems like this was a controversial take, which I find pretty interesting.
I believe people took this comment as LLMs being better than security aware engineers who have the time to spend building solid systems.
This wasn't the point, the reality outside "established" tech companies is that software security can be lackluster.
There often simply isn't enough resources to check old software for basic vulnerabilities, outdated packages with known issues, there might be a manager who insists on a certain solution. Or that certain services are on "maintenance mode", but rarely get checked since they're on a certain part of the internal network.
If they are able to properly scan their full software stack for CVE issues, they cannot deal with a flood of CVEs.
----
We're in a transition period where AI will eventually make software much more secure than it ever was.
These noisy CVEs will probably lead to agents verifying vulnerabilities before humans review them.
The problem with agent reviews from what I can think of is:
- cost to use LLMs to review things
- not necessarily easy to plug-and-play in repos: (domain knowledge + vulnerability knowledge)
- especially with anthropic: able to use models defensively, without hitting guardrails
The last one is the most interesting one to me. How does the AI providers know if you're a "good or bad" guy? And does it matter if open source models is catching up?
We're in a kind of cyber arms race wether we like it or not.
Currently we either get AI promo vulnerability dumps like from Chrome with pretty graphs that no one checks or false positives.
Where is this one now that was hyped everywhere?
https://news.ycombinator.com/item?id=49133889
The GitHub submitter could no longer reproduce the issue and the LKML post has no replies:
https://lore.kernel.org/all/CALCETrXbj__SFQMzPZhES5y6-sh4np-...
All the big maintainers I know of disagree. Curl for example is getting overwhelmed with real issues that LLMs are finding. A year ago most LLM found issues where false positives (the code is correct), or hallucinations (the code in question doesn't even exist) - but that has changed and now LLMs are finding real issues in code. Many different projects have reported the same rules.
Absolutely. I'm not sure how universal this is, but I saw George Hotz tweeted something about Mythos not being such a big deal since he's able to find security issues without a claimed $20,000 in token spend per vulnerability.
The reality is that humans aren't perfect, so we do not make "perfect programs".
It is also possible to test it yourself to see how good LLMs are at finding software issues:
Find any open source project that isn't huge, then have Kimi 3 review it for potential issues.
You will find code "issues".
Most of the time it won't be detrimental for security, but often there are code paths that are not "accounted for", or you're able to bypass a type of check by interacting with the software in a strange way.
So we just need a redundant array of George Hotz, I wonder how much that costs?
This is where quantity becomes a quality. Actors with huge budgets and at least a small team to keep the AIs on track now have an immense amount of potential power. Anthropic has somewhat resisted in giving state actors like this full access to 'dangerous' models, but it appears other companies like OpenAI have no such reservations.
Before such states may have difficulty hiring enough security researchers/hackers. Now you can spin them up. Someone with a few data centers of GPU compute and enough proxies could probably blitzkrieg the internet in a day and capture ungodly amounts of information.
Not just the cost, also boredom. I have used AI to trace down some tricky multi-threaded issues (someone has to write that cross thread framework, and it was me - fortunately most of our code isn't this complex). I already knew the code well, but it would have taken me a long time to find the same issue - the more likely thing is I would give up. Even if I don't give up I'm likely to get bored/frustrated. A month worth of tokens replaced what I would have needed a year to do if I was allowed to spend a year on it.
I'm sure if somebody hired him for a pentest it'd cost more per vulnerability found.
I think his point was more that he didn't believe it was difficult to find these issues.
Could be Dunning-Kruger, but if AI models continue to get better this trend will surely continue.
1. George Hotz exhibits strong signs of Dunning-Kruger in most things he does nowadays.
2. Even in exploit development, his original field, he might be able to find vulnerabilities in mature, well-maintained software with security programs. But would he charge $20k or more for that service? Say it with me, Claude: “Absolutely!”