> while you could ask your LLM to do that you aren't going to run your bank on the result.
Why not?
I feel like we're entering a new era of prejudice against not a category of humans, but against non-human intelligences.
The design patterns for distributed and fault-tolerant systems are well-known and established in the industry. Both humans and AIs are familiar with them!
So if you sketch a design for the AI to follow, establish the rules in AGENTS.md, have a robust test suite, use a frontier model dialed up to eleven, etc... why not rely on the LLM output?
At the end of the day, humans are not without fault either.
I've been wading through some legacy "pre-AI" code recently and it has more bugs than a rainforest! Static fields used incorrectly, causing data races. Floating point types used for money amounts. JavaScript and SQL injection up the wazoo. Wildly unsafe password handling. So on, and so forth. This is the norm for most human-written software, not the exception.
As a proof-of-concept, I tried an AI rewrite of one such legacy app[1], and it is not bug free, but it notably has fewer bugs than the original. Different bugs, sure, and I'll have to iron them out after a round or two of UAT, but I'm honestly more confident with what I got from the chatbot than the code inherited from humans.
[1] Deals with money, but admittedly at a much lower level of risk and consequence than a banking app running on a mainframe.
Because you know that the current one works. If you have a bank running on COBOL (or whatever), you've had that for 30+ years now, so while it might have bugs, you know what they are. You don't know what the LLM output is. Hence back to my original point: writing the code is not the hard bit. Making yourself (and your CEO etc) comfortable to put that into production is one of the hard bits.
> Because you know that the current one works.
What do you even mean by "works", specifically?
> it might have bugs, you know what they are.
Okay, so it doesn't work, you know it doesn't work, it's just that you accept the specific ways in which it doesn't work.
I've lost track of all the myriad stupid ways in which these ancient systems are hugely ineffectual without even being outright faulty.
Like airline tickets where your name is printed as "LASTFIRSTMR" in all caps and no spaces because their systems are ancient beyond belief.
Similarly, my bank statements are security-critical, because anyone with a copy of my credit card details can pull money out of my account without my express authorization. But...
... because they're stored in terrible ancient mainframe databases, the text fields all have tiny maximum lengths. Hence they're all abbreviations. Attacker-controlled abbreviations without any authenticity assurance of any kind!
I have no idea who actually transfers money out of my accounts! There are no URLs, no metadata, nothing to actually confirm the identity of the other party. Every field in a transaction record is 100% attacker-controlled and unverified by my bank.
If you look at it from the perspective of someone used to modern web security, then you realise that banking is a raging tyre in comparison. Banks literally just accept a certain rate of criminal activity and "price that in", reversing transactions when asked -- which itself can also be a criminal activity. They just shrug their shoulders.
"What can we do about this?" -- says the people that have tried nothing and are all out of ideas.
Rewrite it. The whole thing.
Use an actual database, something made in the last three decades instead of half a century ago.
Use cryptography. No, not crypto coins! I just mean a bog-standard algorithms like public-private key signing so that it is possible to confirm the source of transactions.
Etc.
I would much rather have something generated with the assistance of a modern LLM than what we have now, which is security holes big enough to drive a panamax container ship through.
How did you conduct this rewrite? Did you hand the AI some specs, some tests, the existing code?
I feel like AI has dramatically changed how complete rewrites can be considered, especially for long-lived, legacy projects.