While I agree that domain expertise has always been a moat, I believe the author is missing something critical: there is a big difference between being able to verify the output of a system is correct, and being able to tell a system how to generate the correct output to begin with.
Personal example: I had a software engineering colleague who was the best coder of financial management systems I've ever encountered. He gained these skills through years of in-the-trenches development. One of the things he told me, and that I also observed, was that the vast majority of financial experts (basically, the people in the accounting department of companies) had an extremely difficult time just telling him what the rules of any particular transaction should be. But what they could do was tell him whether the handling of any particular transaction was right or wrong. So often times he would sit down with these accounting folks and go through lots of example transactions he came up with, and from there he essentially built up the requirements spec.
In my experience, that is the primary difference between people I've known who are good software engineers and those who aren't: people who can specify the detailed rules of any system, vs. folks who take a "well, I know it when I see it" approach.
I have a strong suspicion that folks who have a high degree of domain expertise in a particular area will fail as software builders even in an agentic world because they will struggle to elucidate clearly the rules in their head that they've learned over years. As an analogy, it's kind of like asking a native speaker for the grammar rules of their language. Often times they can't, but they'll just say "well, that sounds wrong." They may be "domain experts" in their language, but they'd have a hell of a time prompting an AI system on how to grade a test for grammar correctness.
Anybody who has ever done programming professionally in the small scale knows this. Refining the requirements is the job.
In fact, I've never known an industry so keen on levelling its own moats as the software industry. We regularly invent things like 4GL, graphical programming and frameworks and engines such as Unity just to enable more people to do programming. People will happily teach programming for free in outreach programs (I am just one example). No other profession does this. Perhaps with the exception of mathematicians and other fields that are very close to programming.
I could teach an economist enough programming in a weeks that they could write an ERP module, but I could not learn enough economics in a week to write one. If the language was the barrier, we could invent a more effective one. We invent new languages weekly anyway. Having seen how quickly beginners can make things with Unity or Godot, I seriously doubt an LLM agent could improve much on this. Of course, if the job is writing yet another CRUD React app with a Java or Python backend, then sure, the LLM will be very effective. But compared to doing same app in something like Excel or MS Access? Not that much.
Everything everywhere does this. All human progress has been making common and accessible the rare and expensive. Universities, online courses, textbooks, etc all exist to make economics as accessible as possible, there’s just a really tight reward loop with programming so your rate of learning it is more tangible
I think you are underestimating how hard it is for average joe to learn programming basics. I remember a fellow in high school that just could not accept that = in programming is assignment not an equation (like in high school math)
I'd like to believe I have an inkling, having done a fair bit of teaching.
Still, imagine how hard other skills are to acquire. How much civil engineering can you learn in two weeks? How much violin playing? But you could absolutely get basic grasps on a general purpose programming language. With something specialized like Unity or Excel you would get tons of useful output.
The hurdle around assignment operator is as old as symbolic languages. That's why legends such as Niklaus Wirth wanted to use another operator, ":=", a notation that is still being used.
Anyway, it can be a hurdle, but one I find that most people get over pretty quickly.
> The hurdle around assignment operator is as old as symbolic languages. That's why legends such as Niklaus Wirth wanted to use another operator, ":=", a notation that is still being used.
Well, sure; but more generally, the ability to accept other meanings for symbols (and keep subtly different symbols straight in one's head) is a mental skill, and individuals vary in their aptitude for it. (Presumably, this one is also relevant to natural language learning, since one must reckon e.g. with false cognates.)
Nah I just think you are so overwhelmed that "=" having a different meaning lh and rh is just too much. It is hard to remember how hard it was.
You went to high school with Niklaus Wirth?
This is one of the few mistakes K&R made going from BCPL to C.
>We regularly invent things like 4GL, graphical programming and frameworks and engines such as Unity just to enable more people to do programming.
You're right on the money on this.
Earlier this month I went to visit a company for a complete demo prototype of a full one-to-one train simulator trainer mostly designed and programmed by a former civil engineer using Unity engine. According to the company, they could not do it if Unity engine (or similar) is not around because it will be prohibitively expensive to develop.
In a related news, Unity recently released AI eco-system namely Unity AI Suite [1].
[1] Unity AI Suite:
https://unity.com/features/ai
the 1 dimensional thinking that's perverse in the industry will be catastrophic.
we don't like doing the hard things e.g training juniors so they can be skilled seniors via good apprenticeship programs i.e on the job. now we r delegating to stochastic parrots.
in terms of systems thinking which is one skill you need to be a domain expertise - very few people are ever curious & are not willing or able to ask critical questions. hence the groupthink that's prevalent in the industry.
no wonder the quality of software never goes up - while the building blocks have gone up in quality. an analogy is like having super strong bricks but making brittle structures
I use LLMs daily for coding. That said, I agree with your sentiment. Super strong bricks, poorly put together, makes a bad building.
Another analogy I like is a beginner playing a $100k Stratavarius probably can't produce anything near a professional violinist playing a $50 violin.
Personally I use LLMs to level up my systems thinking. I describe the domain, I have it brainstorm some scalable solutions, I look them up, I bring them up to the team, and we discuss, and I implement. It's a great workflow imo.
>One of the things he told me, and that I also observed, was that the vast majority of financial experts (basically, the people in the accounting department of companies) had an extremely difficult time just telling him what the rules of any particular transaction should be
We have internalized more knowledge than we can explain sounds like the textbook definition of Polanyi's paradox:"Polanyi's paradox, named in honour of the British-Hungarian philosopher Michael Polanyi, is the theory that human knowledge of how the world functions and of our own capability are, to a large extent, beyond our explicit understanding" [0]
[0] https://en.wikipedia.org/wiki/Polanyi%27s_paradox
Thanks so much! I hadn't heard of Polanyi's paradox before but it is exactly what I was talking about. The Wikipedia entry even highlights the problem for AI-driven development:
> Polanyi's paradox has been widely considered to identify a major obstacle in the fields of AI and automation, since programming an automated task or system is difficult unless a complete and fully specific description of the procedure is available.
having worked on automation of accounting process, it has been a combination of ever evolving ruleset/edge scenarios for 10% cases plus standardised rules for 90% of the cases. i think the problem is that it is an evolving field. alot of times it borders law, which makes it murky and vague too.
[dead]
cf https://en.wikipedia.org/wiki/Dual_process_theory as a causative agent behind the paradox.
> there is a big difference between being able to verify the output of a system is correct, and being able to tell a system how to generate the correct output to begin with.
There are attempts to avoid needing to explicitly construct the rules. This is often called "Programming by example".
https://en.wikipedia.org/wiki/Programming_by_example
On some level, even training an LLM to answer questions is this. I do like my determinism, though.
Vibecoding with human-managed acceptance is a very current-moment way of doing this. Just make sure the agent has to program within a framework, not changing it, and the results are actually pretty good.
One of the interesting things one can do with this mentality, if you set up your system well, is to reevaluate prior decisions with new rules, and decide whether any differing decisions are corrections for prior bad decisions, or newly-introduced bugs.
I think this neatly captures the gap in the article; it’s thinking of domain expertise as static, rather than something that needs to be built/extracted.
It also goes the other way; A a good engineer can build experiments and discover the domain even without an oracle on the team; reading protocols or manuals or specs, for example. In other words, learn to be a domain expert themselves.
Personally I find the most efficient way to learn a concept to be building it; you are immediately faced with your blind spots. AI for sure helps improve cycle time on these discoveries if you use it that way; time writing boilerplate goes to ~zero and you can spend your time figuring out how to answer the real questions.
I do think that we as a profession need to learn new architectural and craftsmanship patterns to make it easier to learn from our code; concepts like Literate Programming which were too fussy for fast-moving teams may end up accelerating in the agentic world; I need to read a lot more code than I write now. But also things like Acceptance Test Driven Design; not new concepts, just newly increased in value.
WRT the native grammar, consider adjective order. Few native english speakers (me included) can off the cuff name the proper order, but everyone knows the "right" order.
https://dictionary.cambridge.org/us/grammar/british-grammar/...
Another one: tekamolo, for German and adverbs this time.
Temporal, kausal, Modal, Lokal
https://www.olesentuition.co.uk/single-post/tekamolo-how-to-...
Excellent example.
Great example
I think the answer is in the second sentence of the article. The most valuable person in this new world (using the author's own phrasing from the penultimate paragraph) is the one who is good at "building a working model of the domain in [their] head".
Depending on the domain, this may either be the domain expert themselves, or someone else trained in formal logic, data structures and organizing information into coherent hierarchies. If this is neither the domain expert nor the programmer, then it must be someone in between. In the old world, this role was called "business analyst".
Working model suitable for an electronic computer.
I'm this person, and I do find AI to be quite helpful, though I'm mostly just playing around at this point.
I'm the daughter and granddaughter of programmers, and I learned the basics of how to code as a kid. I'm good at it and have a knack for it, but I didn't want to do it for 8+ hours a day and then spend my nights on it as well, so I didn't pursue it as a career. I did an undergraduate degree in Linguistics, which has been really helpful for having an intuitive sense for what 'language as data' can accomplish and for a strong understanding of the difference between language as data and language as meaning. I studied formal logic systems. Then I did a graduate degree in Library Science and worked in libraries for a decade and a half.
I can organize and define systems very well, and I'm trained in how to wheedle information people don't consciously know out of them without them knowing I'm doing it. I've spent enough time around actual devs to understand where my limitations are and when to loop in someone who knows more to check my work, and when it's important for the work to be super accurate versus when I can learn by fucking around. (Front end and design? Fuck around! Database structure? Fuck around but with an exceptionally robust backup system kept outside of the AI tools' purview + don't fuck around in prod. Storing credentials and people's information? Ask someone.)
The problem companies are going to have is I'm very disinclined to work for them doing this, particularly if they want us because they think we're going to be cheaper. Most people who are in this category a.) could be devs and opted not to, and there's a reason for that and/or b.) are the children, cousins, etc. of programmers. We're not stupid: we know we're just as disposable as they're trying to make devs.
> which has been really helpful for having an intuitive sense for what 'language as data' can accomplish and for a strong understanding of the difference between language as data and language as meaning.
100%. I think, intuitively, software developers understand that there's a strong connection here, but most fail to translate that into practice. It always amuses me when someone comments on the triviality of creating CRUD apps. Setting aside the fact that people usually get the mechanics of it wrong (despite it being a solved problem), they overlook the difficulty of producing a good information design.I've developed software in a variety of industries, and I would say maybe 5% of the designs I inherit are well-done and represent the concepts they're trying to model in an elegant, parsimonious way. Rather, most examples are replete with ambiguity, orthogonal concepts smashed together into single elements, and misleading naming and relationships.
This is so true.
And even when there is a well laid out information design, it often is laid out from the wrong point of view. Elegant information design should create a pattern subconsciously enabling people to build a mental model of the tool they're using without realizing they're doing it. But software whose information design works wonderfully from the point of view of a SDE is not going to be approachable to the average user. There are so many tools I've used where I can very clearly see the reasoning behind the decisions and the design and use it well and also be aware I could never explain how they work to the average person.
Having descended from a humanities social background and blundered into professional programming rather incidentally, a lot of this resonates with me.
I've frequently been credited as a person who can really string all the disparate elements of tacit knowledge together into a unified fabric in our particular subdomain, and helped a lot of people plug Swiss cheese gaps in their knowledge that way and come away with the feeling that it's all been tied together theoretically.
However, it's not immediately obvious to me how, in our LLM psychosis cultural moment, this facility shoots to the top of the value chain.
In theory, it's because we're going to be better at steering an LLM in some ways. A lot of the friction and hold up in building comes in the communication between people/departments/organization. If you eliminate that by holding the knowledge in one person/department, you see an efficiency gain.
What they're not saying is that they think we're more valuable because they think we'll be cheaper. They think they can have us do 2 jobs and pay us for 1: probably less than a decent SDE made in 2019.
Personally, I think it's likely to be a shitshow and backfire if that's how companies decide to try to go that route (especially the largest ones). First, if we wanted to be devs, we would be (like you). Most people with the knack for programming and thinking in systems know they have that knack, and if they haven't jumped ship to SDE before now, there's a reason. I could definitely hack a junior SDE role, skill wise, but I don't want to. Second, finding people like us is difficult. The hiring process (which is becoming more and more Gilliamesque by the day) is really bad at identifying us. There aren't credentials, and this sort of work tends to reside in the gaps, as you identified. It's harder for it to show up on a resume. Hiring is optimizing for exact matches and experience, and that's the opposite of how this skill set actually functions. I've found these skills are best developed by being placed in a room where you know very little about what's going on and forcing you to develop heuristics and approaches over time for getting that context. Thirdly, I can't speak for you, but I've developed this perspective over decades and if people want it, they're going to pay appropriately. If they think I'm going to do any of this at my current salary level, they're deranged. And lastly, while most people in our position might roll our eyes at some techie discussions and culture, we do fundamentally like techies/devs and we tend towards placing a greater value on things like relationships than a pure SDE does. (Just speaking in generalities). So 'is willing to replace and/or toss out a category of people I like and respect' is a hint to us to start out assuming this is a hostile negotiation. (Whereas SDEs as a cohort over the last 20 years extended a lot of goodwill at first). We're far more likely to work somewhere, get enough domain knowledge, and then bounce to start our own thing, especially since as a population we're more likely to have devs who will work with us as non-technical founders. Someone who's decent at marketing/sales/the stupid 'people stuff', understands a domain, and understands when a proper dev tells them what is and isn't possible and can even help with some of the most boring, rote parts of the technical side if needed/in crunch is an excellent non-technical founder, and as a group we're also more likely to have access to the technical connections that we'd need if we wanted to build something beyond our ability.
This is an underappreciated bit of insight and perspective, and I thank you for sharing it.
Thoughts that really stood out for congruence with my own experience:
> Most people with the knack for programming and thinking in systems know they have that knack
> Hiring is optimizing for exact matches and experience, and that's the opposite of how this skill set actually functions.
> I've found these skills are best developed by being placed in a room where you know very little about what's going on and forcing you to develop heuristics and approaches over time for getting that context.
> We're far more likely to work somewhere, get enough domain knowledge, and then bounce to start our own thing
Having worked with domain experts I concur on the difficult time they have expressing the rules of their domains.
Once I built a little domain-specific language for them, that was tested against old jobs to see if they contradicted the past; it was a nifty project and since then I am convinced that DSLs are underrated as a way to encode expertise.
Can you give an example (even if contrived) of how that would have looked ? I’m very curious !
I had the experts write markdown files that contained the rules looked somewhat like:
## 1A Rule name
Some prose explaining the rules liking to official documentation.
``` if municipality and inhabitants > 10000 then functionA else functionB ```
Then a trivial parser would extract the rules, the DSL was then handled by Lark[1]. So pretty simple, but it made collaborating with experts easier as simulated results would also output some markdown they could read.
1. https://lark-parser.readthedocs.io/en/stable/
Sounds like "Literate Programming", where the code and comments are reversed: instead of everything being code except what's marked as a comment, everything is a comment except what's marked as code.
I am also very interested!
Scala to the rescue, then! :)
I agree with your point that people “from” the domain aren’t automatically equipped to start AI-building software in the domain, exactly because they often lack one of the most crucial software development skills — being able to (and having the desire to) describe a complex system with a finite set of deterministic rules.
But I don’t think we should be calling these people “domain experts”. I think we should reserve that name for the other group, for the people who truly and deeply understand the domain, the whys and whats and why nots.
> But I don’t think we should be calling these people “domain experts”. I think we should reserve that name for the other group, for the people who truly and deeply understand the domain, the whys and whats and why nots.
I've spent a lot of time in my career extracting info from the business and I think most do understand the whys/why nots but aren't practiced at organizing all of those decades of experience into a higher level abstract model that can easily be communicated.
It's typically layers and layers of information with dependencies in many directions littered with exceptions. Just like our software design+dev experience, it takes a lot of practice to try to organize all of that info into a coherent presentable model.
That ship has sailed. Anyone who works in a field is now considered a domain expert (or "SME") even if they're nothing of the sort. There should ideally be another term that's a superset, but I doubt it would ever catch on.
Then there are almost no domain experts.
Agree. If you can't explain it to someone else, you don't really understand it yourself. These people are practitioners not experts.
You're doing a bit of a "true Scotsman fallacy" here.
It feels like without a bit of true Scotsmanship, the term “domain expert” doesn’t really mean anything.
Sure it does, it’s just a highly contested meaning that depends on a proper contemporary interpretation if to be received as intended in this context. Context that varies domain to domain. For the domain name buyer it’s quite different than the metacognitive scholar.
People you say are not experts are fully capable to explain whys and whats and why nots. In more details that you can absorb.
They will do that on examples tho. They will recognize and explain every single exception they see - but they are not able to list all exceptions up front. Because, that is highly unusual task
I think that you raise some excellent points here. I want to dive deeper into what is meant by domain knowledge -- as a I see it. This phrase comes to mind: "Those who know, do. Those that understand, teach." (Google search tells me that it is frequently misattributed to Aristotle.) People who cannot clearly articulate rules for their domain only "do". However, those that can clearly articulate will "teach". Thus, I would say true domain knowledge is the ability to teach that domain to others.
I largely agree, but want to note that there’s the counterpoint “Those who can, do; those who can’t, teach” (from George Bernard Shaw’s 1903 play Man and Superman), which also has some nugget of truth in it.
Case on point is Youtube where a lot of almost good amateurs give a lot of advice on niche channels on things they hardly master.
> So often times he would sit down with these accounting folks and go through lots of example transactions he came up with, and from there he essentially built up the requirements spec.
Right, so the spec is derived from examples, an interactive process that doesn’t require a programmer.
Interpolating between the examples still requires understanding of the domain for the interpolation to be a sensible one. And it’s an iterative feedback process: thinking about possible interpolations leads you to cases that need clarifications from the domain experts. The domain experts won’t come up with all relevant examples by themselves, and they typically aren’t as good at thinking about interpolations like a developer who is trained to always consider all possible cases; they don’t think in terms of models the way a developer does.
LLMs already have interactive discussions with me on the topics I engage with, including asking expansive questions, so I do not think this is beyond the realm of the technology.
Talking to CPAs never gets you to Pacioli groups.
Talking to engineers and mathematicians gets you there.
The programmer skill is how to abstract the specs from all the examples. And then to formalize it. Actual coding is merely translation. And only beginners tend to focus on that.
Sounds very similar to what AI is good at.
I've yet to see AI be good at extracting good abstractions. More often than not it jumps to creating a battery of special case checks.
No. AI sucks at formal languages and implications.
Lossy decompression probably won’t ever be really good at this. If it’s missing from the data, then it won’t be there.
The AI math proofs were probably already out there in tiny pieces and nobody got all of the pieces together. That is valuable. It’s different from making a piece that is missing. And the AI will just hallucinate.
May work for simple cases, but how would you come up with something like typst, or thunderbird?
Yes. The author stumbled on the p vs np problem but with humans. Verification is easy. Solving is the millennium prize question.
I agree that it has some resemblance, but the striking thing about NP-complete problems is that an efficient solution to any of them is an efficient solution to all of them, which makes it worth trying exceptionally hard to find one.
It could be that whatever lackluster expertise you can squeeze out of an LLM is good enough to discourage investment in the real thing since unlike NP-complete problems, expertise isn't generalizable.
Agreed. But there is a new dynamic coming to place. Who can execute faster now that we have AIs? I think the combination of (Good SWE using AIs) collaborating with (Domain Expert using AIs) is the winning team. Things are accelerating on all fronts, the frontier may be jagged but AIs is making progress on all capabilites.
Additional iterations are valuable, if and only if we learn from additional iterations. Otherwise we are just spinning our wheels.
> But what they could do was tell him whether the handling of any particular transaction was right or wrong. So often times he would sit down with these accounting folks and go through lots of example transactions he came up with, and from there he essentially built up the requirements spec.
This combination of requirements is the business in most domains. Software or otherwise. Codefying different rules and executing them.
This is the exasperating part about learning to speak Spanish using a textbook; you must guess the grammar rules because the textbook won't tell you. So, you use the English rule and hope and pray that it is the same in Spanish, and you'll be right the majority of the time but often wrong. Spanish textbooks written 100 years ago tell you the grammar rules and are more useful than recent textbooks.
While interesting, I rarely found knowing grammar rules beyond some very basic ones to help all that much for learning to speak a language, compared to language exposure and speaking practice.
I’ve noticed this too. Any idea why modern language books get this so wrong?
I have found this with (mechanical) engineers. They know what they want to see but don’t understand the underlying details which are more mathematical than the average engineer is able to work with. So the people working on engineering software are often physicists or applied mathematicians.
Ran into this, as well.
And they were really annoyed at being asked math questions.
> I have a strong suspicion that folks who have a high degree of domain expertise in a particular area will fail as software builders even in an agentic world because they will struggle to elucidate clearly the rules in their head that they've learned over years.
This is the part I disagree with.
In a non-agentic world, the expert and the programmer are two different people. If the expert finds a bug in the software, they have to describe that bug, send it over to a programmer to fix, wait for a new release and until that scenario occurs again, realize that their description was actually wrong, send a corrected description, rinse and repeat for a few iterations until the bug is finally fixed for good.
In a world of agents, the expert finds the bug, asks the agent to fix it, realizes that the fix is incorrect because of sloppy thinking, does a few iterations until the feature works correctly, and that's it. Bug fixed; with 10 minutes of work instead of a few weeks.
> realizes that the fix is incorrect because of sloppy thinking
If the domain expert doesn’t understand the generated code, they can only discover incorrect logic by specific examples (specific inputs), which is usually impossible to do exhaustively. The programmer, on the other hand, can see incorrect logic directly, generalized over all possible inputs and states. It’s the difference between testing and proving correctness.
> In a world of agents, the expert finds the bug, asks the agent to fix it, realizes that the fix is incorrect because of sloppy thinking, does a few iterations until the feature works correctly
That would require clearly (a) knowing what you want, and (b) expressing it unambiguously and in detail, including all edge cases. Essentially producing a spec.
Most people are not able to do either. Talking to an LLM does not change that.
but.. they can iterate real quick!
As long as the expert don't run of patience, he may be able to do that.
Consider finance sector -- the industry is powered by excel spreadsheets made by non-programmer having no clue how programming works.
Spreadsheet is a DSL for finance and a lot of other sectors. But the lack of formal logic capabilities shows when you seen the spreadsheet organization. Most only stand with ducktape all over the place.
I had the same problem in air travel industry. Our ticketing officers were never able to communicate how the ticket commission rules from our partners should be interpreted. I ended up adding a test framework, that allowed to add examples and counterexamples of existing flights to each rule, to verify that they are applied the way they'd assign commission by hand.
Domain expertise matters more than most bootcamp grads want to hear. Knowing how to phrase a query for a specific domain is half the battle.
You described reasoning by analogy, which two thirds of the population does.
Please explain explicitly what you are implying, because I don't get it
Who are the 1/3 of the population that does not reason by analogy?
Instead of needing a familiar metaphor to understand a new system (reasoning by analogy), they grasp the core conceptual mechanics directly.
It’s not that the 2/3 can't think abstractly, it’s just that they use analogies as a temporary processing step to get there, while the 1/3 starts there.
I fully agree here. It's also not uncommon to have the reply for specific items be "Ahh, well that is a bit of an edge case see because of X and Y" and then you learn there are many many "special cases" to the point that they aren't that special anymore.
This comment brings back memories of a project.
We were working through many complex business flows and running into exactly what you describe in many areas. Some of the project people with less experience complained that we were spending too much time on exceptions and slowing down the project. We had to explain that when each exception happens 100 times a day with significant impact on business productivity, then it doesn't matter whether you call it an exception or not, it's important to solve for.
That is why coders who learn problem domains are where the money is at.
Works for me.
Already learned two or three problem domains well enough and now it is starting to compound.
This is basically the gap BDD tries to close I guess. It assumes that domain experts can't fully articulate the rules upfront.
Some companies use product managers for this purpose. They translate the implicit domain knowledge of those experts into explicit requirements, mostly through interviewing.
> I have a strong suspicion that folks who have a high degree of domain expertise in a particular area will fail as software builders even in an agentic world because they will struggle to elucidate clearly the rules in their head that they've learned over years.
Maybe they need a build a hybrid expertise of "domain" and "software engineering". For example, robotic surgery requires expert surgeons to build sufficient expertise in robotics
Also, noticed a pretty high karma for a throwaway account.
I very much agree here--the true job of us programmers is determining exactly what the problem is. The domain expert will fail because they haven't attempted to refine the details enough, they probably don't have the skill to anticipate the edge cases.
I've spent most of my career working with the same people--even people who have some coding experience but they never catch the low probability stuff. Or even not so low probability: business procedures were that the customer was quoted an all-inclusive price. Try to work backwards, what is the pre-tax amount? Um, that's not guaranteed to have a solution. My concerns get ignored, I go ahead and implement with code that will catch the offending penny and label it roundoff correction. While I'm not sure what happened I think some auditor hit that. I ended up having to walk them through the calculation, okay, what's the answer here? Only when they couldn't solve the problem could they accept that what I had done was the only answer. (The actual odds of such a failure are equal to the tax rate. I had not originally worked that out, just saw success was not guaranteed.)
The domain experts make this sort of mistake routinely in dealing with code. AI won't fix that.
Are we thinking about the wrong side here? Wouldn't the LLM be a super-intelligent financial expert as well as a super-intelligent coder?
It is neither.
I guess the flip side to that is the iterative conversation to develop these specifications is exactly what AI chatbots are good at.
It's also the main reason why very structured AI agent orchestration for software engineering modelled on rigid processes fails to really provide much value.
I suspect that the reason AI chatbots are not very good at developing such specifications is that they don’t build a domain model in their “head” — the only models they have are already hardcoded in their weights. A human learning a domain, on the other hand, forms new neural connections representing the model of the domain that they build in their head.