I really don't know how we are going to deal with accelerated exploit generation via LLMs — it blows up the disclosure processes we've evolved.
The only thing I can think of is hunkering down with static HTML to reduce attack surface, but that's laughably incompatible with many many business requirements.
In new software I write these days I limit the number of external dependencies and mostly vibe code anything I need besides the most critical foundational libraries (increasingly I am replacing these as well).
I think this, the supply chain attacks and binary reverse engineering will cause us to go back to thin clients, html over the wire and vibe coded libs instead of external open source deps.
Vibe coding deps is security through obscurity, with all of its weaknesses (and strengths). It's not that such deps have fewer security bugs than the open source reference implementations from which they are derived, it's just that those bugs are currently less discoverable.
In the near to medium term, the current open source regime has issues because bugs have become extremely discoverable and easy to exploit. But I am skeptical that obscurity will provide vibe coded deps sufficient protection over the long term. As soon as the source code for those vibe-coded deps leaks, you are screwed.
It's not just obscurity, it's increasingly difficult to trust the governance and quality of code of any outside library. A lot of deps in code are unnecessary you use maybe 1-10% of the api surface. But since building and maintaining software can be cheaper now it can be better to maintain a 1-5k LOC vibe coded version that meets your security bar than to trust an external party's governance & quality. This has the additional benefit of reconsidering if you even need the dep or if you can reuse something you already have (since you may not want to maintain the lib).
I really doubt that. If these models continue to improve, they will be producing higher quality code than, maybe not the best human developers, but certainly the average human developer who would have typically been writing code on these systems.
Pre-2024 code is full of human-written vulnerabilities.
Limiting the number of dependencies is always good, within reason. Especially with eco-systems that want to pull in everything and the kitchen sink. That's just asking for a supply chain attack. I try to limit myself to what well reputed distros provide and what I can review. That's still no guarantee but it cuts down the risk substantially.
I think it will kill open source to some degree. I found it already very hard to imagine how the endless queues of issues were being dealt with so diligently, but at least there was a human on the other side. The torrent that is unleashed now is not manageable on a human scale. It's a highly automated arms race with a bunch of good-willing humans caught in the crossfire between large companies and criminals duking it out.
A lot of people in the open source world actually enjoy the act of programming. Not all of them will enjoy prompting to the same degree, but to keep pace with the bad guys you will have to either join or find something else to do. That's going to be a big loss.
It's funny, that's true although I personally can't relate — I enjoy having built something robust, but the act of building isn't something I love.
I can imagine prompting an LLM to build provably correct software. Aren't we already doing something similar already when we remind it to write unit tests and account for all edge cases?
Actually writing provably correct software from scratch sounds terribly tedious, which is why I've never explored it. But maybe an LLM could make that task more approachable.
I switch between the two modes. I love algorithmic stuff and I really don't like working on boilerplate and plumbing. So I use AI for the second and read every line and all the other stuff I roll by hand. The speedup isn't super like that but the amount of control is pretty good.
I just gave this prompt to duck.ai (5.6 Terra under the hood): "How would one go about writing a provably correct algorithm to count UTF-8 code points?"
It gave me something which looked pretty legit. Of course, the generated output very likely to be subtly wrong, but as starting input to a formal checked specification/proof process it seemed like a good start.
I wrote some UTF-8 based string handling software back in the day. It doesn't seem unreasonable to write up a set of requirements and edge cases and then have a formal mechanism prove correctness for common UTF-8 string manipulation operations.
[PEAK HN ON] Now if we could just write provably correct exception handling, we could solve one of the two PHP vulns from today. [PEAK HN OFF]
UTF8 is one nasty little standard. So many edge cases.
I should check it against my email address validator that would be a nice benchmark. It's interesting that there are people that are using email addresses that are not actually valid according to the standard.
We can have an LLM enumerate all those edge cases, and we can assemble them into a spec. Then we can integrate that spec into a proof.
And because the spec is functional rather than expressive, we can produce copyright-free, provably correct reference implementations whose correctness is as strong as our spec and our proof process.
This reference implementation would then be available for use by a language like PHP, or perhaps an LLM generating a vibe-coded dependency.
Over time, then, the accumulation of such proven reference works grows. We'll never prove all the application code out there correct, but incremental progress is possible.
I think such constructs should be part of the language or at least of the standard runtime library and not re-implemented by every Tom, Dick, Harriet & HAL.
Prompting the LLM to "look for vulnerabilities in PHP's multipart upload handling code" is going to find stuff that "look for vulnerabilities in PHP" won't.
I was more talking about staying up on security issues with your stack after they are patched. We will need to check for security updates constantly, or risk being exploited in the gap between a security release and you finding out that you need to update your software.
Yes, so who will pay for that? Open source so far meant you would use your brains and your creativity, now you need to use your money too if you want to keep pace. From a 'guts, credits & glory' proposition to 'slop processor if you can afford it' is not really an upgrade.
I'm already tired, between pushing back on vibe coded internal projects being put onto the internet and then the constant barrage of CVEs that mean nothing.
Then you have real security incidents that you need to tune into constantly.
It's going to be a rough decade.
I really don't know how we are going to deal with accelerated exploit generation via LLMs — it blows up the disclosure processes we've evolved.
The only thing I can think of is hunkering down with static HTML to reduce attack surface, but that's laughably incompatible with many many business requirements.
In new software I write these days I limit the number of external dependencies and mostly vibe code anything I need besides the most critical foundational libraries (increasingly I am replacing these as well).
I think this, the supply chain attacks and binary reverse engineering will cause us to go back to thin clients, html over the wire and vibe coded libs instead of external open source deps.
Vibe coding deps is security through obscurity, with all of its weaknesses (and strengths). It's not that such deps have fewer security bugs than the open source reference implementations from which they are derived, it's just that those bugs are currently less discoverable.
In the near to medium term, the current open source regime has issues because bugs have become extremely discoverable and easy to exploit. But I am skeptical that obscurity will provide vibe coded deps sufficient protection over the long term. As soon as the source code for those vibe-coded deps leaks, you are screwed.
It's not just obscurity, it's increasingly difficult to trust the governance and quality of code of any outside library. A lot of deps in code are unnecessary you use maybe 1-10% of the api surface. But since building and maintaining software can be cheaper now it can be better to maintain a 1-5k LOC vibe coded version that meets your security bar than to trust an external party's governance & quality. This has the additional benefit of reconsidering if you even need the dep or if you can reuse something you already have (since you may not want to maintain the lib).
> it's increasingly difficult to trust the governance and quality of code of any outside library
The value of 'code written before 2024' is going to be substantially higher than of code written afterwards.
Sort of like a 'low background steel', a comparison someone made on HN before which I think was pretty good.
I really doubt that. If these models continue to improve, they will be producing higher quality code than, maybe not the best human developers, but certainly the average human developer who would have typically been writing code on these systems.
Pre-2024 code is full of human-written vulnerabilities.
This only applies to its value in regards to training data for LLMs.
No, it will also be reflected in the value of IP.
Limiting the number of dependencies is always good, within reason. Especially with eco-systems that want to pull in everything and the kitchen sink. That's just asking for a supply chain attack. I try to limit myself to what well reputed distros provide and what I can review. That's still no guarantee but it cuts down the risk substantially.
I think it will kill open source to some degree. I found it already very hard to imagine how the endless queues of issues were being dealt with so diligently, but at least there was a human on the other side. The torrent that is unleashed now is not manageable on a human scale. It's a highly automated arms race with a bunch of good-willing humans caught in the crossfire between large companies and criminals duking it out.
Could open source activity transition towards provably correct software, with far more modest feature sets?
I don't have experience with proof based tools, so I'm not sure what's feasible.
A lot of people in the open source world actually enjoy the act of programming. Not all of them will enjoy prompting to the same degree, but to keep pace with the bad guys you will have to either join or find something else to do. That's going to be a big loss.
It's funny, that's true although I personally can't relate — I enjoy having built something robust, but the act of building isn't something I love.
I can imagine prompting an LLM to build provably correct software. Aren't we already doing something similar already when we remind it to write unit tests and account for all edge cases?
Actually writing provably correct software from scratch sounds terribly tedious, which is why I've never explored it. But maybe an LLM could make that task more approachable.
I switch between the two modes. I love algorithmic stuff and I really don't like working on boilerplate and plumbing. So I use AI for the second and read every line and all the other stuff I roll by hand. The speedup isn't super like that but the amount of control is pretty good.
I just gave this prompt to duck.ai (5.6 Terra under the hood): "How would one go about writing a provably correct algorithm to count UTF-8 code points?"
It gave me something which looked pretty legit. Of course, the generated output very likely to be subtly wrong, but as starting input to a formal checked specification/proof process it seemed like a good start.
I wrote some UTF-8 based string handling software back in the day. It doesn't seem unreasonable to write up a set of requirements and edge cases and then have a formal mechanism prove correctness for common UTF-8 string manipulation operations.
[PEAK HN ON] Now if we could just write provably correct exception handling, we could solve one of the two PHP vulns from today. [PEAK HN OFF]
UTF8 is one nasty little standard. So many edge cases.
I should check it against my email address validator that would be a nice benchmark. It's interesting that there are people that are using email addresses that are not actually valid according to the standard.
But UTF-8 is also extremely well-studied.
We can have an LLM enumerate all those edge cases, and we can assemble them into a spec. Then we can integrate that spec into a proof.
And because the spec is functional rather than expressive, we can produce copyright-free, provably correct reference implementations whose correctness is as strong as our spec and our proof process.
This reference implementation would then be available for use by a language like PHP, or perhaps an LLM generating a vibe-coded dependency.
Over time, then, the accumulation of such proven reference works grows. We'll never prove all the application code out there correct, but incremental progress is possible.
I think such constructs should be part of the language or at least of the standard runtime library and not re-implemented by every Tom, Dick, Harriet & HAL.
Business requirements will have to change. All your competitors are under the same pressure with the same shitty set of options, after all.
That would have to start with businesses shelling out more funds for the packages that their businesses depend on.
It already _is_ a rough decade. I have hated the 20s so damn much.
Yeah, the future of our jobs in this industry... for the type of work we do. are concerning. Explosure has whole new meaning
It feels like the only solution is going to be proactively running your own AI agents looking for patches and applying them.
Does any one know of good feeds that we should be pulling or scraping from? I think the proactive nature here is seriously important.
The issue is where to direct the LLM's attention.
Prompting the LLM to "look for vulnerabilities in PHP's multipart upload handling code" is going to find stuff that "look for vulnerabilities in PHP" won't.
I was more talking about staying up on security issues with your stack after they are patched. We will need to check for security updates constantly, or risk being exploited in the gap between a security release and you finding out that you need to update your software.
Yes, so who will pay for that? Open source so far meant you would use your brains and your creativity, now you need to use your money too if you want to keep pace. From a 'guts, credits & glory' proposition to 'slop processor if you can afford it' is not really an upgrade.
Or, you know, use client certificates so the bots can’t hit you.
Those can leak / be stolen of course, but a bunch of my clients php apis survived as without a cert they’re told to fuck right off.
It does make their client onboarding a bit harder but it’s a trade off which seems to have been the right one.
This doesn’t help you if you need to run a public website.
[dead]
I'm already tired, between pushing back on vibe coded internal projects being put onto the internet and then the constant barrage of CVEs that mean nothing.
Then you have real security incidents that you need to tune into constantly.
job security!