I guess at some point we will have lawyers, attorneys and judges using this stuff ... at the point lawyers will become kinda "seo"/"copywriter" experts on how to better trick the others LLM.

Almost makes me want to get a law degree.

I mean, the laws are written down somewhere though. A human can still look at the actual law and surmise that the AI is feeding them bullshit.

I think the problem is that laws overlap, with decades of case law clarifying their interactions. Looking at one law probably isn't enough to determine whether an LLM is lying to you.

Human judges today often don't bother to look at "the actual law" and surmise that the human is feeding them bullshit.

the look on the face of the Court administrator upon hearing someone describe the "paperclip maximizer" problem.. ominous!