>One of the things he told me, and that I also observed, was that the vast majority of financial experts (basically, the people in the accounting department of companies) had an extremely difficult time just telling him what the rules of any particular transaction should be
We have internalized more knowledge than we can explain sounds like the textbook definition of Polanyi's paradox:"Polanyi's paradox, named in honour of the British-Hungarian philosopher Michael Polanyi, is the theory that human knowledge of how the world functions and of our own capability are, to a large extent, beyond our explicit understanding" [0]
Thanks so much! I hadn't heard of Polanyi's paradox before but it is exactly what I was talking about. The Wikipedia entry even highlights the problem for AI-driven development:
> Polanyi's paradox has been widely considered to identify a major obstacle in the fields of AI and automation, since programming an automated task or system is difficult unless a complete and fully specific description of the procedure is available.
having worked on automation of accounting process, it has been a combination of ever evolving ruleset/edge scenarios for 10% cases plus standardised rules for 90% of the cases. i think the problem is that it is an evolving field. alot of times it borders law, which makes it murky and vague too.
[dead]
cf https://en.wikipedia.org/wiki/Dual_process_theory as a causative agent behind the paradox.