Because you know that the current one works. If you have a bank running on COBOL (or whatever), you've had that for 30+ years now, so while it might have bugs, you know what they are. You don't know what the LLM output is. Hence back to my original point: writing the code is not the hard bit. Making yourself (and your CEO etc) comfortable to put that into production is one of the hard bits.
> Because you know that the current one works.
What do you even mean by "works", specifically?
> it might have bugs, you know what they are.
Okay, so it doesn't work, you know it doesn't work, it's just that you accept the specific ways in which it doesn't work.
I've lost track of all the myriad stupid ways in which these ancient systems are hugely ineffectual without even being outright faulty.
Like airline tickets where your name is printed as "LASTFIRSTMR" in all caps and no spaces because their systems are ancient beyond belief.
Similarly, my bank statements are security-critical, because anyone with a copy of my credit card details can pull money out of my account without my express authorization. But...
... because they're stored in terrible ancient mainframe databases, the text fields all have tiny maximum lengths. Hence they're all abbreviations. Attacker-controlled abbreviations without any authenticity assurance of any kind!
I have no idea who actually transfers money out of my accounts! There are no URLs, no metadata, nothing to actually confirm the identity of the other party. Every field in a transaction record is 100% attacker-controlled and unverified by my bank.
If you look at it from the perspective of someone used to modern web security, then you realise that banking is a raging tyre in comparison. Banks literally just accept a certain rate of criminal activity and "price that in", reversing transactions when asked -- which itself can also be a criminal activity. They just shrug their shoulders.
"What can we do about this?" -- says the people that have tried nothing and are all out of ideas.
Rewrite it. The whole thing.
Use an actual database, something made in the last three decades instead of half a century ago.
Use cryptography. No, not crypto coins! I just mean a bog-standard algorithms like public-private key signing so that it is possible to confirm the source of transactions.
Etc.
I would much rather have something generated with the assistance of a modern LLM than what we have now, which is security holes big enough to drive a panamax container ship through.