LLMs certainly make it more feasible to rewrite a product in a memory-safe language, eliminating a whole class of bugs.
Flawless software is hard for an LLM to write, because all the programs they have been trained on are flawed as well.
As a fun exercise, you could give a coding agent a hunk of non-trivial software (such as the Linux kernel, or postgresql, or whatever), and tell it over and over again: find a flaw in this, fix it. I'm pretty sure it won't ever tell you "now it's perfect" (and do this reproducibly).
The CVEs here have their fair share of silly C problems, but also more rigid input validation and handling. These more rigid validations exclude stuff which may even be valid by the spec, but entirely problematic in practice.
As examples, take a look how many valid XML documents are practically considered unsafe and not parsed, for example due to recursive entity expansion. This renders the parsers not flawless and in fact not in spec.
Or, my favorite bait - there should be a maximum length limit on passwords. Why would you ever need a kilobyte sized password?
Have you ever met a security engineer? I’ve never met one who was also a good engineer (not saying they don’t exist, I just haven’t met one). Do they find vulnerabilities? Sure. Could they write the tools they use to find vulnerabilities, most probably not.
Just because something is good at finding bugs, it may not find all the bugs. Finding a bug only tells you there was one bug you found, it doesn't tell if the rest is solid.
Who said it can't? https://news.ycombinator.com/item?id=47759709 appears to be a nearly flawless (per spec) zip implementation.
the AI found no bugs in AI code
sure buddy
Because the problem is asymmetric: the attacker only needs to find one hole at one time. The defender has to be flawless forever.
It’s easier to break something than it is to make something that cannot be broken.
LLMs certainly make it more feasible to rewrite a product in a memory-safe language, eliminating a whole class of bugs.
Flawless software is hard for an LLM to write, because all the programs they have been trained on are flawed as well.
As a fun exercise, you could give a coding agent a hunk of non-trivial software (such as the Linux kernel, or postgresql, or whatever), and tell it over and over again: find a flaw in this, fix it. I'm pretty sure it won't ever tell you "now it's perfect" (and do this reproducibly).
If humans can find bugs, why can't humans write flawless code?
Whatever the answer to that conundrum might be, LLMs are trained on these patterns and replicate them pretty faithfully.
How do you define flawless though?
The CVEs here have their fair share of silly C problems, but also more rigid input validation and handling. These more rigid validations exclude stuff which may even be valid by the spec, but entirely problematic in practice.
As examples, take a look how many valid XML documents are practically considered unsafe and not parsed, for example due to recursive entity expansion. This renders the parsers not flawless and in fact not in spec.
Or, my favorite bait - there should be a maximum length limit on passwords. Why would you ever need a kilobyte sized password?
Have you ever met a security engineer? I’ve never met one who was also a good engineer (not saying they don’t exist, I just haven’t met one). Do they find vulnerabilities? Sure. Could they write the tools they use to find vulnerabilities, most probably not.
Just because something is good at finding bugs, it may not find all the bugs. Finding a bug only tells you there was one bug you found, it doesn't tell if the rest is solid.
You could argue the answer to this question depends on if you believe P=NP