Genuinely wondering (aka not snarky): Has anyone found frontier models to provide useful research in the context of European civil law systems?
Your comment made me wonder if there are any halfway-acceptable model benchmarks for law tasks? Specifically I’d love to know how the frontier models’ abilities compare between common law vs. civil law systems. My guess would be that an AI in a common law context should have a clearer idea of how a specific case is interpreted/accepted by (common law) practitioners, whereas trying to rely on AI in a civil law context, like Germany, can be daunting. In a few Germany-specific recent examples, the models feel like they present only (maybe too stubbornly?) the “civil law”-based laws. All while negating much of AI’s research benefits because civil statutes are portrayed as being absolutely accurate, binding, and their enforcement (and thereby the legal reality) being uniformly applied. Am I making this interpretation up? If so, how can I prove myself wrong?
I don't know of anything that is working yet, but I know some people working on it (intentionally vague).
Roman law, on which the legal systems of Germany, most of Europe, Turkey etc. are based relies more heavily on statutes than Anglo-American case law, but cases do play a role there, too. That's why a practically useful system also needs to have access to court decisions, commentary etc. - and while the statues are in the public domain, a lot of the other knowledge is owned by specialist publishers (such as: C.H. Beck, Nomos, Mohr Siebeck, De Gruyter, Otto Schmidt, C.F. Müller, Manz, Verlag Österreich, Schulthess, Stämpfli, Dalloz, LexisNexis, Lefebvre Dalloz, Giuffrè Francis Lefebvre, CEDAM, Giappichelli, Il Mulino, Aranzadi, Tirant lo Blanch, Marcial Pons, Dykinson, Tecnos, Lefebvre, Almedina, Gestlegal, AAFDL, Kluwer, Boom juridisch, Wolters Kluwer, Larcier-Intersentia, Anthemis, Sakkoulas, Nomiki Bibliothiki, Universul Juridic, Hamangiu).
Internationally, legal systems appear to undergo a sort of convergence, which means statutes increasingly matter in the U.S. and cases increasingly matter in European law (according to a law partner friend).
So those lawyers will likely want to know about the likeliness of it, strategies to work around it, and overall mitigations.
It might be helpful if the company launching this product included such information in their blog post instead of ignoring it in a field with such a high cost of getting it wrong.
agreed, but i dont think there are many (any?) product launch blogs that cover all the fine details and legal minutia of the product.
that all typically comes in pages of terms of service, purchase contracts, SLAs, conversations with your rep, knowledge base articles, and that sort of thing. not in the initial marketing post.
Does hallucination matter for this application? We've moved beyond raw recall being that important, it seems like for law specifically all relevant facts will be cited and checked easily by humans.
> all relevant facts will be cited and checked easily by humans
I've talked to a lawyer about how they handle this. They do indeed double-check everything, since it'd be embarrassing (or worse) to send hallucinated statements to opposing council or to the court. They still find the assembly a huge time saver
But based on stories in the news on the subject, not everyone has this same level of diligence
Sure but just like generating 100x more code, someone has to review it. So you are wasting everyone in court's time (defendants, prosecutors, judges, staff) by making them parse through what is quite often a bunch of hallucinated slop. Time that could be much better spent on parties who prepared and reviews their own arguments.
The lawyers I know are very fixated on the problem of hallucinated case citations which is amusing to me as a onetime programmer, since case citations have a well-defined syntax and would be relatively easy to check programmatically.
Which makes it all the more bizarre that LLMs have this problem. Claude Code runs the code it generates through a compiler, why can’t an LLM run its product through a cite checker? I’ve seen LLMs fabricate citations.
Not like common law doesnt already have a lot of hallucination going on.
can “penumbras and emanations” compete with hallucinations?
Common law is all about rummaging around in dead mens’ letters, LLMs are a natural fit.
Genuinely wondering (aka not snarky): Has anyone found frontier models to provide useful research in the context of European civil law systems?
Your comment made me wonder if there are any halfway-acceptable model benchmarks for law tasks? Specifically I’d love to know how the frontier models’ abilities compare between common law vs. civil law systems. My guess would be that an AI in a common law context should have a clearer idea of how a specific case is interpreted/accepted by (common law) practitioners, whereas trying to rely on AI in a civil law context, like Germany, can be daunting. In a few Germany-specific recent examples, the models feel like they present only (maybe too stubbornly?) the “civil law”-based laws. All while negating much of AI’s research benefits because civil statutes are portrayed as being absolutely accurate, binding, and their enforcement (and thereby the legal reality) being uniformly applied. Am I making this interpretation up? If so, how can I prove myself wrong?
I don't know of anything that is working yet, but I know some people working on it (intentionally vague).
Roman law, on which the legal systems of Germany, most of Europe, Turkey etc. are based relies more heavily on statutes than Anglo-American case law, but cases do play a role there, too. That's why a practically useful system also needs to have access to court decisions, commentary etc. - and while the statues are in the public domain, a lot of the other knowledge is owned by specialist publishers (such as: C.H. Beck, Nomos, Mohr Siebeck, De Gruyter, Otto Schmidt, C.F. Müller, Manz, Verlag Österreich, Schulthess, Stämpfli, Dalloz, LexisNexis, Lefebvre Dalloz, Giuffrè Francis Lefebvre, CEDAM, Giappichelli, Il Mulino, Aranzadi, Tirant lo Blanch, Marcial Pons, Dykinson, Tecnos, Lefebvre, Almedina, Gestlegal, AAFDL, Kluwer, Boom juridisch, Wolters Kluwer, Larcier-Intersentia, Anthemis, Sakkoulas, Nomiki Bibliothiki, Universul Juridic, Hamangiu).
Internationally, legal systems appear to undergo a sort of convergence, which means statutes increasingly matter in the U.S. and cases increasingly matter in European law (according to a law partner friend).
Why fix an unsolvable problem, when you can demo DOCX generation and make just as much money?
The benchmark you linked to shows GPT-6 Astra having the lowest hallucination rate of all tested models.
The blog post makes no mention. If the rate isn't 0% it should be mentioned for a field like this.
Agreed. Hallucinations and reliability are the main hurdles to anything being 'agi' in my book
The definition of AGI is whatever a frontier lab wrote a blog about doing in the last week.
why this maximalism? There's nothing that has 0% hallucination including humans. lets use reasonable baselines.
Humans can be sanctioned and fined, and eventually disbarred if they continue to lie in court filings.
presumably that will remain true for the lawyers using the product
So those lawyers will likely want to know about the likeliness of it, strategies to work around it, and overall mitigations.
It might be helpful if the company launching this product included such information in their blog post instead of ignoring it in a field with such a high cost of getting it wrong.
agreed, but i dont think there are many (any?) product launch blogs that cover all the fine details and legal minutia of the product.
that all typically comes in pages of terms of service, purchase contracts, SLAs, conversations with your rep, knowledge base articles, and that sort of thing. not in the initial marketing post.
probably available by the "contact legal sales" link at the top of the post https://openai.com/business/contact-sales-legal/
Does hallucination matter for this application? We've moved beyond raw recall being that important, it seems like for law specifically all relevant facts will be cited and checked easily by humans.
> it seems like for law specifically all relevant facts will be cited and checked easily by humans.
Don't be too sure about that. [0]
0: https://www.damiencharlotin.com/hallucinations/
> all relevant facts will be cited and checked easily by humans
I've talked to a lawyer about how they handle this. They do indeed double-check everything, since it'd be embarrassing (or worse) to send hallucinated statements to opposing council or to the court. They still find the assembly a huge time saver
But based on stories in the news on the subject, not everyone has this same level of diligence
Sure but just like generating 100x more code, someone has to review it. So you are wasting everyone in court's time (defendants, prosecutors, judges, staff) by making them parse through what is quite often a bunch of hallucinated slop. Time that could be much better spent on parties who prepared and reviews their own arguments.
The lawyers I know are very fixated on the problem of hallucinated case citations which is amusing to me as a onetime programmer, since case citations have a well-defined syntax and would be relatively easy to check programmatically.
Which makes it all the more bizarre that LLMs have this problem. Claude Code runs the code it generates through a compiler, why can’t an LLM run its product through a cite checker? I’ve seen LLMs fabricate citations.
A modern frontier model with an agentic harness will check its citations if the system instructions or prompt tell it to.
You can get an effectively-zero hallucination rate with the right setup already.