I would think that LLMs would be better at avoiding foot-guns. That’s a situation where you have a list of well known rules and potential pit falls, and the work of the lawyer is to apply those to a fact pattern. That’s something that has been hard to automate programmatically, because the fact patterns are similar but different. LLMs, however, seem to excel at applying general principles to differing fact patterns.
Instead, the LLMs create entirely new foot guns like citing non-existent cases. You can't go more than a week without encountering another news report of a lawyer submitting an AI-generated legal brief rife with bogus case citations, which even includes briefs submitted to state supreme courts.
e.g., https://www.npr.org/2026/04/03/nx-s1-5761454/penalties-stack...
I would categorize this in the "expertise that people internalize but never figure out how to verbalize" department, and that is a department we have no way to teach an LLM because if nobody is writing out those unspoken, subconscious rules then the LLM has nothing to read about them in its training data.
This is often called tacit knowledge. https://en.wikipedia.org/wiki/Tacit_knowledge
My favorite example of this is knowing how to untangle a big pile of cables. There are robots now which can untie a single knotted cable, but I don't think any can do a pile of cables yet. https://www.youtube.com/watch?v=vp-94rsherE
> and that is a department we have no way to teach an LLM because if nobody is writing out those unspoken, subconscious rules then the LLM has nothing to read about them in its training data.
I think on the contrary, LLM providers accumulate huge logs of interaction with their users, which elicit that tacit knowledge and mine it and humans cooperate willingly in order to solve their tasks. Just imagine the corpus of sessions for scientific research, education or software development, it is probably the largest such collection ever to exist. Trillions of HITL tokens per day flow into those logs, carrying our perspectives, choices, original ideas and tacit knowledge. I call this the "human-AI experience flywheel". It's the new stackoverflow, next model generation is based on interaction data from previous one.
Good point. Same probably applies to code as well, coders much tell us why they write the cde the way they did. And if they have comments in their code, those are highly untrustworthy because noboy fixes comments if the code works.
I don't know the source off hand, but I've seen llms hallucinating case citations in order to "prove" their premises.
can't get more foot gun than "well according to [fiction] it is a well established practice (that the defendent is guilty)"
But can an LLM come up with questions like what the definition of is is? Seems to me there's a lot of "depends on how you read it" type of stuff that lawyers excel at finding novel interpretations. So what coders thinking of as rules are much less straight forward to understand when it comes to laws
I think that’s a different task than the one OP is referring to. To your example, I’m not familiar with the capability of LLMs in that regard. I have struggled with using the AI features of westlaw when it comes to that sort of argument. (Basically, making an argument that strays from typical route, because that’s the position you happen to find yourself representing.)
I'd only be guessing, but I'd imagine that trying to simulate being a lawyer for someone trying to do something shady would really push an LLM. Imagine being a lawyer for Trump. Could it ever come up with the arguments that his lawyers have? God help us all if they do