The problem with LLM's is that they could work correctly for months and years and then do something egregious which will will go unnoticed because of the misplaced trust one develops on a system that "just seems to work." Get flagged for an expensive audit and there go all the savings and then some.
isn't that true for humans too?
Humans are more likely to make small mistakes but the internal consistency check is pretty good at catching large errors. On top of that, fudging numbers to make everything add up is not something humans do (not unintentionally at least)