If you're not already applying static analysis and linters to your codebase (and I know many of you aren't), ask yourself why you would bother to apply an expensive LLM tool?
Not to say these things won't catch vulnerabilities static tools cannot, I think they can, it's just we already have the capability to automatically catch a large surface area of common vulns, and have chosen not to, often for expense reasons.
If you're a team that does already apply several layers of analysis and linting, and wants to add this on top, all power to you.
> If you're not already applying static analysis and linters to your codebase
Because most issues are in business logic that static analyzers aren't going to catch.
If you run a static analysis tool across a repo that didn’t previously do that, you’ll see that while what you say might be true, there’s going to be an absolute treasure-trove of issues caught by the static analyser.
Static analysis won't develop a one click exploit that works end to end for you.
I'm at a FAANG and even our static analysis tools are not great at identifying how many issues are actually reachable.
Ideally you use both. An AI model that has static analysis as part of the harness, so it can evaluate each potential finding.
> Ideally you use both. An AI model that has static analysis as part of the harness, so it can evaluate each potential finding.
Ideally the static analysis tools are improved so that we don't need to piss away yet more tokens like we're competing on Mark's leaderboard just to find vulnerabilities.
When you reach that ideal world, let me know. My company has thrown a decade+ and multiple teams at the idea you've described. We still aren't there yet.
Your proposal of relying purely on static analysis is over-idealistic and just not feasible for large, diverse codebases in the real world.
That's where AI comes in.
> Your proposal of relying purely on static analysis is over-idealistic and just not feasible for large, diverse codebases in the real world.
"Just not feasible" is thought terminating, but regardless, I thought we were talking about ideals? Ideally you want the static analysis to work, not to rely on the non-deterministic bullshitter.
> piss away yet more tokens
> non-deterministic bullshitter.
You're so ideologically opposed to AI that you bury your head in the sand in cases where it genuinely does a fantastic job today, right now, in the real world (like developing end to end exploits using noisy signals like static analysis results, fuzzer results, etc).
Instead you assert that we should go a route no company has successfully proven out despite throwing billions of dollars and some of the best cybersecurity talent in the world at.
Anyways, if you develop a static analysis solution that works across large, diverse production codebases and develops end to end working exploits without AI, I will literally buy it off you for millions of dollars. Or you could start your own company. You'd be an overnight decabillionaire.
I actually do use AI, I wouldn't say I'm ideologically opposed lol. Maybe I'm ideologically opposed to thought terminating clichés, or how FAANGers see it as a cudgel to cram in wherever we find an open gap just to shit infinite tokens into?
You just haven't suggested a single solution that achieves the same level of risk reduction as AI driven end-to-end exploit generation.
You claim static analysis does the job, but you haven't backed it up with any proof that it works across large diverse codebases. Meanwhile, we have proof that AI works at least somewhat, here and now.
Static analysis often shows many false positives. A more intelligent tool can help not to waste limited engineering time.
False positives are noise, but if the tool is filtering out its own noise via AI, it might work. Or you could take a high false positive/low false negative tool and instead of bothering humans with its noisy output, have AI investigate and evaluate if found issues are false positives or not.
I quite like that the most honest answer for the majority of devs was downvoted then flagged to death.
Most people doing this now didn't use static analysis tools because they were seen as an unnecessary extra.
[dead]