No discussion on this can avoid mentioning this ongoing disaster...
"AI Hallucination Cases" - https://www.damiencharlotin.com/hallucinations/
No discussion on this can avoid mentioning this ongoing disaster...
"AI Hallucination Cases" - https://www.damiencharlotin.com/hallucinations/
When I first tried using IDE AI coding assistants a bit over a year ago, they would frequently fail to read files and completely hallucinate the code I asked it to analyze or edit. And even when it did successfully read my code, it would often hallucinate the API of other frameworks.
I don’t think hallucinations are a major issue with agentic coding a year later? At least, not for me.
Legal writing does have the problem that there is no such thing as a correct argument that a goal-driven loop can target. But basic things like “does this case even exist” seem solvable with research agents, and even harder problems like “does this case support the argument being made” seem conceptually very similar to questions I now ask Claude daily.
Yes, mechanical question of "does this exist" is trivial to solve. Whether the research is actually aligned with the user intent / the AI makes claims that are substantiated by the research at hand are much harder problems.
> I don’t think hallucinations are a major issue with agentic coding a year later? At least, not for me.
Are you still looking at code and agent traces to know if this is true or not?
At work, we still see hallucinations across all model families as we share bot claims for each others code
An issue I come across is the attributed failures of AI of a first time use of AI.
Non-deterministic software can behave orders of magnitude different based on how input is structured.
This factor is largely missed by bringing a deterministic mindset to trying AI.
This means the same problem solution can work for one person reliably, and not another and both are true.
Assertions that begin with a deterministic assessment using a snapshot in time of both the user and models capabilities are a new feature of this new category of software that will have to be better caught and expressed.
I provided some very light guidance to a couple of masters students who were working on legal citation hallucination detection last summer. Their summer project is here: https://github.com/gt-csse/mellea-lrc
If anyone here is looking for engineers to work on legal AI products, both of the students who worked on this project come highly recommended.
There being multiple cases every single day is nuts.
But also, interesting how the rate of cases peaked and seems to be going down now.
To your second point, it seems like every time a significant practical limitation is identified, it goes through a familiar pattern. It gets a news cycle, and then it gets a wave of confidently incorrect opinionating, and the problem gets treated as if it's a permanently built in problem that's baked into the foundation of AI. And then time passes and the issue is either mitigated or circumvented entirely.
Hallucinations are one, "AI can't scale anymore" is another (remember that one?), limited context windows, strawberry problems, glue on pizza, sycophancy.
I don't want to say there's no problems, there are very serious ones. User psychosis, economic harms, alignment, cost of computer components, etc. But there's a lot of false positives that get turned into Ed Zitron articles along the way.
So many of these we likely using Google to search then found AI mode and used that. Which is to say, they were (and still are!) using Google Gemini 1.5 from many years ago.