Eventually there will be an incident with bad software at a hospital or bank that leaves some people dead or broke.

Then regulators will take things seriously.

This is exactly what Uncle Bob predicted in his talk "The Future Of Programming" [0] 10 years ago, way before LLMs.

[0] https://www.youtube.com/watch?v=ecIWPzGEbFc

Which is why the medical device software industry is so heavily regulated after the Therac-25 incident. Oh, wait, it's not.

https://en.wikipedia.org/wiki/Therac-25

What regulators?