The fact that software quality plays zero role in commercial software engineering is one of the reasons why I think LLMs will easily eat our lunch. Bugs simply don't matter.
The fact that software quality plays zero role in commercial software engineering is one of the reasons why I think LLMs will easily eat our lunch. Bugs simply don't matter.
I would have disagreed with you in the past by saying, "until it breaks something critical and you loose customers and business", but then again people just moved on from the Crowdstrike incident like business as usual.If something like that which grounded critical service globally and had an estimated 10 Billion Dollar economic impact doesn't change mindsets,I don't know what will.
That's because no one died. All the safety critical industries are already heavily regulated. E.g. check out for example standards like DO-178C (for software in airborne systems), where you even have to _prove_ correctness of every tool and dependency you use, on top of accountability and traceability of every single line of code in your own product.
Not to be pedantic, but people have died from software programming bugs being a primary contributing factor. One example: Therac-25 (https://en.wikipedia.org/wiki/Therac-25)
I only meant this in relation to Crowdstrike incident that was mentioned in the comment I replied to. The standards and regulations in those other industries have changed dramatically (for the better) since Theract-25.
I mean, that was over 40y ago. Same thing for the Ariane 5 failure which is a staple of safety-critical classes (at least in Europe), it's not getting any younger.
If all the examples you can conjure are decades old*, is it any wonder that people don't really take it seriously? Software power the whole world, and yet the example of critical failure we constantly hear about is close to half a century old?
I think the more insidious thing is all the "minor" pains being inflicted by software bugs, that when summed up reach crazy level of harm. It's just diluted so less striking. But even then, it's hard to say if the alternative of not using software would have been better overall.
* maybe they've added Boeing 737 Max to the list now?
> If all the examples you can conjure are decades old
They're not ALL the examples I can conjure up. MCAS would probably be an example of a modern software bug that killed a bunch of people.
How about the 1991 failure of the Patriot missile to defend against a SCUD missile due to a software bug not accounting for clock drift, causing 28 lives lost?
Or the 2009 loss of Air France 447 where the software displayed all sorts of confusing information in what was an unreliable airspeed situation?
Old incidents are the most likely to be widely disseminated, which is why they're most likely to be discussed, but that doesn't mean that the discussion resolving around old events mean the situation isn't happening now.
In aviation, accidents never happen because of just a single factor. MCAS was mainly an issue in lack of adequate pilot training for this feature, AF447 was complete incompetence from the pilots. (the captain when he returned to the cockpit, quickly realized what was happening, but it was too late)
There's almost never a death where there is a single factor, regardless of aviation or not. You can always decompose systems into various layers of abstractions and relationships. But software bugs are definitely a contributing cause.
Another way to put it is that people have to literally die before a company is taken to task over their product's quality. What a sad, low bar for a product!
What about the less obvious ways people are dying. Teen suicide rates have been directly linked to social media, for example.
And memory leaks (one of the main gripes of TFA) aren't even a thing because you cannot use malloc in this sort of code.
People died in Boeing 737 mcas incidents and what? Hot fix and business as usual.
Unless bug results in enormous direct financial loses like in Knight Capital, the result is the same: no one held responsible, continue business as usual.
> and had an estimated 10 Billion Dollar economic impact
This might be more to do with these estimates than anything.
Bugs don't matter once you have captured your userbase.
Until then they matter a lot.
One thing that needs to be addressed is how easy it is to build moats around your users which trap them in your ecosystem. It's great from a business perspective if you can pull it off, but it's killing innovation and making users frustrated and apathetic towards technology as a whole.
Considering that LLM are fairly good at finding security relevant issues - the bugs that matter - there is certainly scope for them. I would not be surprised if not checking your code with an LLM will be considered negligent in the near future. This week I was forced by circumstances to triage a messed up nginx configuration. While not the focus of my work the LLM pointed out two issues where configuration was not following best practices with security relevant implications (it also turned out an old release was used by the team and one issue was already fixed due to the feedback of a pen test assessment).
LLM seem to be really good in analyzing things. I don't not trust them to produce too much but alone the ability to take a few files and bits and pieces and ask for a response with a certain direction has been transformative to my work.
LLMs are not a silver bullet. They're just as likely to point out non-issues, as they are to hallucinate fixes for them.
Their output is only as valuable as the human using them. If they're not a security expert to begin with, they can be easily led astray and lulled into a false sense of security.
See curl, for example. Hundreds of bogus reports rejected by an expert human. One large report with valuable data, that still requires an expert human to sift through and validate.
Bugs matter. LLMs aren't going to eat our lunch on their own. They'll do so in service to some human that's using them to exploit a bug.
People have been developing neural networks since the 70's, and there's been two major hurdles to their usefulness for software development:
1. They require gigabytes to terabytes of training data.
2. A non-trivial percentage of output data is low confidence.
The first problem requires tens to hundreds of gigabytes of training data.
This first problem not only requires the slow but predictable increase in processing power and data storage capabilities that were unachievable until recently, but is also only possible because open-source software has majorly caught on, something that was hoped for but not a given early in AI development.
The second problem means that the output will be error prone, without significant procedural processing of the output data that is a lot of work to develop. I never would have thought that software writing by neural networks would be competitive, not because of effective error control, but because the entire field of software development would be so bad at what they do (https://xkcd.com/2030/) that error-prone output would be competitive.
Why? If software quality doesn't matter to us, why would it matter to an LLM?
If it's trained on our code, how would it know any better?
If it's driven by our prompts, why would we prompt it to do any better?
You can't be serious.
How do you categorize "commercial software engineering"? Does a company with $100M+ ARR count? Surely you can understand the impact that deleting a production database can have on a business.
And your answer to this is "LLMs will eat our lunch" and "bugs don't matter"? Unbelievable.
Any business that isn't prepared for the possibility of losing their production database and doesn't have a plan for recovering from that is not a serious business.
The database incident was a single example.
Do you really think that most businesses are prepared to handle issues caused by their own bugs, let alone those caused by the software they depend on? That's nothing but fantasy. And your attitude is "they deserve it"? Get real.
No, I never said they deserve it.
But if you have a business, and don't have continuity and recovery plans for software disasters, that's like not having fire insurance on your facility.
Fire insurance (and backups/disaster recovery plans) doesn't mean there won't be disruption, but it makes the disaster survivable, whereas without it your business is probably ended.
And losing a database or a major part of one is as simple as one adminstrator accidentally running "drop database" or "delete from customers" or "rm -rf" in the wrong environment. It happens, I've helped recover from it, and it doesn't take an AI running amok to do it.
> No, I never said they deserve it.
That's the gist of your argument. They're not a "serious business", therefore it's their fault. Let's not mince words.
> It happens, I've helped recover from it, and it doesn't take an AI running amok to do it.
Again, losing a database is not the issue. I don't know why you fixated on that. The issue is that most modern software is buggy and risky to use in ways that a typical business is not well equipped to handle. "AI" can only make matters worse, with users having a false sense of confidence in its output. Thinking otherwise is delusional to the point of being dangerous.
Bugs are another reason you have backups. There are any number of ways you can lose data. Hardware can fail. Electrical surges can fry your computer.
If your business depends on that data, then you take steps to protect it. Or not, at your peril.