How do you handle the fact that an LLM can't seem to help itself from disgorging page after page of words no matter what it's asked to do? I've never seen an LLM say "this looks good as-is; I would not spend any more time on it; what's next?" it will always seem to suggest using another pattern or additional abstractions or other yak shaving.
But to be fair human code reviews have the same problem. It's like reviewers feel they have not done their job if they don't find something wrong.
I've totally had latest models just say "here are minor nits but this is good to ship" when reviewing code. Claude is a bit more verbose usually but Codex by default is pretty terse. My experience anyways. They also take pushback on suggestions (this isn't actually a problem because X) and they'll agree and say ship it (hopefully only when you're right and it's not a problem ;) ).
I personally rely on this behavior of LLMs. For example, for reviewing prose, I might ask the LLM to find five issues. If there are obvious issues I missed, then it'll find them. But if all the issues it lists are minor or hallucinated, then I can have some confidence there weren't any glaring issues.
I do the same for LLM code review comments: some changes are out of scope or could be moved to a separate PR; some edge cases don't happen in practice and should just fail noisily instead of writing more code to maintain. When these are the only issues it's raising, then I know it's done.
I've tried and tried to get an LLM to delete code, it couldn't do it. I knew a file was 40% bad so what I ended up doing was deleting the file, then asking AI to create the missing file. That's how I got AI to delete code :)
Sounds like the AI got you to delete the code for it.
It is surprisingly hard to do in a single prompt. I’ve had good luck though with being very explicit about asking for review, then to think deeply about what actually matters, and then reduce to a very short extremely concise reply. —- the model NEEDS to output those pages of text as part of its thinking process. It’s used to doing it in the assistant reply but can be cajoled into doing it in the thinking tokens.
Yeah, that's just a problem with review. I tend to keep it going until the stuff its finding is stuff that I look at and think "I can live with that". Usually by that point it's found the worthwhile stuff and what remains is minor or, like you said, yak shaving.
There used to be a bit of "wisdom" passed around that said "when you submit something to the client (or management, etc) for approval, leave an obvious mistake somewhere." Then they will find it and point it out, you can easily correct it, and everyone is happy. Otherwise they will find something to critique, just to demonstrate that they contributed something to the review.
I wonder if the same trick works with AI?
I think so, if you can inject synthetic errors with the same distribution as the errors you care about.
What tools are you using? I've found that Codex will happily tell me there's nothing more to do after a few rounds of /review and fix (or sometimes, it picks up on low value points that are technically correct but not worth changing).
My guess would be that this is a matter of fine-tuning the model to the problem. You would need to run the review experiment N times, observe the distribution of answers, and then fine tune with model operational parameters.
Perhaps someone more knowledgable could jump in here to clarify?
I’ve found stating “only call out critical issues, report but ignore everything else” will basically get what you want. Put that in a review skill and you’re good to go.
To be clear I mostly only use Opus and Gemini Flash but this might work for others too.
> I’ve found stating “only call out critical issues, report but ignore everything else” will basically get what you want.
Disabling your C compiler warnings works too. You get to ship then leave work early!
The verbosity itself seems to be a problem with post-4.6 Claude rather than a general issue with all LLMs. and IME the yak shaving is due to an overly generic prompt. We have a generic automated review bot but it’s prompted to only look for errors and never suggests refactoring; that was a conscious trade off to avoid what you’re talking about. If you’re reviewing interactively then you can give additional guidance about any code smells etc. that jump out