I think this is probably true for most skilled professions. AI is best used in the hands of folks already knowledgeable in the skills/professions they are using it for.
I liken it to me googling things as a sysadmin vs. Jane from accounting doing it. The non-tech end user is far more likely to make the problem worse, or install something sketchy from the ad riddled results than I am, or one of my help desk employees are.
I wouldn't trust myself to draft an important legal document using AI without the advice of a lawyer, much like I wouldn't really want to rely on my lawyer to use AI to write code for me.
I find those that are best and make the greatest use are the ones who remain skeptical but also use the tool. The same people who were already nuanced and picky before AI. The same people who already doubted and questioned their own work, and used that suspicion to help prevent them from having over confidence in their own work. If you weren't willing to just "lgtm" with your own code, it's difficult to do that with AI.
(To be clear, I'm not saying perfectionists. Some might call them that because the picky people have higher standards, but a good expert has to also understand that perfection doesn't exist. That's often a driving force in the suspicion! This also tends to cause them to continually improve)
I would agree with this point and as I explained in a comment replying to the GP comment above, that atrophy is far more dangerous in the legal field than it is with code because legal documents do not benefit from the structural safeguards available for code, like automated testing, static typing, static analysis tools, etc. IME with legal LLMs so far, they are easily in that most dangerous valley where they can lull you into a false sense of security while still introducing extremely dangerous mistakes that are frequently difficult to detect without very careful reading.
The danger of those mistakes creeping in also grows exponentially the farther a lawyer strays from their core legal expertise. There are a few statutes I know inside and out, and I can spot LLM analytical errors related to them in a split second, but once I venture out into domains where I am not an expert (but where I am nevertheless reasonably qualified to practice), it becomes much harder to spot drafting mistakes because I have not refreshed my own understanding of the law by reviewing the relevant cases or statutes as I would when drafting the analysis myself from scratch.
> I agree, BUT I also find that it's easy for experts to atrophy quickly. When the AI is right 80/90% of the time it lulls you into over confidence
Thinking the AI is right 80/90% of the time is already a sign of being lulled into overconfidence. The actual percentage is much lower in my experience. I'm willing to grant the AI is "somewhat right" that often but is that really what we settle for?
Am I secretly the only person who ever actually cared about being very accurate. Is AI just an excuse everyone else is using so they can stop pretending? This is so incredibly frustrating
> If you weren't willing to just "lgtm" with your own code, it's difficult to do that with AI.
If you are willing to do that with your own code you should probably not be trusted to work on software
> I wouldn't really want to rely on my lawyer to use AI to write code for me.
Yet that is exactly what a lot of C-Suiters (many of whom are lawyers), are doing.
Vice versa there is also a lot of irresponsible programmers doing stupid things with ai. Irresponsible people stay irresponsible, AI just make them more productive at being irresponsible.
The problem is the low levels have no influence whatsoever. The higher ups force crap down and none ever comes back.
Corporations are DEMANDING legal ai because it is so much more efficient.
Lawyers creating legal stuff, via LLMs is OK. Programmers creating software through LLMs is OK.
Mixing them, is, not, in my experience, OK. In the future, I am sure that LLMs will reach the point, where their output will be beyond reproach, but we're not there, yet.
That means that someone that knows the context and content, needs to vet the output, before sending it on.
> In the future, I am sure that LLMs will reach the point, where their output will be beyond reproach, but we're not there, yet.
I have no doubt that you're right, but will it be because they are close to infallible or because we have let ourselves become lazy and reliant?
My money is on lazy and reliant based on the trends I'm actually seeing
> sysadmin
Another domain where LLMs are very effective at confidently leading people down a messy path. I have a roommate using LLMs to guide him through setting up some ollama stuff in my WSL (I happen to have the half-decent GPU here) and after multiple rounds of the bot trying to get him to do things that were redundant if not in the wrong direction entirely (and vaguely insulting as a matter of course), I had to write "ground truths" along these lines, and probably more as I find them:
[Yes, it did that] [roommate + bot spent 45 minutes on trying to configure their way through NAT when not having to do that is almost the entire point of tailscale. It was just (essentially) like, "You're absolutely right. We have tailscale set up, so we don't need to be able to ssh to that other interface at all. Not troubleshooting that would have saved 45 whole minutes. Oh well, now what?"]Maybe it's just me, but I'm not inclined to trust the judgment of something that can't keep this kind of thing straight, which I know is to some degree a matter of having all the needed info in the context window. But maybe it would be able to do that if it didn't waste tokens telling me to cd into the same directory that I'm already in every 2 minutes, or chmod .ssh/ again, or (when it really needs to burn some tokens) blow away the .venv and pull a bunch of modules again just to "start clean".
im not so sure
i think devs overestimate their own role and underestimate others
i am seeing lawyers and doctors roll out their own software with AI
but we dont have their training and experience
So a software engineer could diagnose an illness with ai, even if they happen to be right that doesn't really prove much about how bad of an idea it could be in a long tail scenario.
Also worth remembering that LLMs have jagged intelligence. They are probably better software developers than anything. Is there a complement to Gell Mann Amnesia- where you assume it’s good at other jobs because it’s good at yours?
Did you not read the article ? Where does it talk about software development/engineering?
It's like that in engineering, for sure. My background is in aerospace and there are lots of things that a reasonably technically-inclined random can probably do passably. It takes an engineer to know which tasks those are, though.
I would imagine it's similar in law, in that it takes a lawyer or judge to know where the foot guns lie.
Agreed, and it's the same in software. Probably the biggest time-sink right now as a tech lead is people going from idea to fully-fleshed-out PR, and then having to go back to have a discussion of "was this the right thing to do". It causes frustration all around (being a "no" much more, and having someone tell you your finished work isn't valuable).