Yeah I personally don’t understand the point of AI code review tools all that much, as AI is already generating the code as well. All of these AI code review tools create so much noise, yet don’t catch the really important things.
Yeah I personally don’t understand the point of AI code review tools all that much, as AI is already generating the code as well. All of these AI code review tools create so much noise, yet don’t catch the really important things.
Thinking of AI-generated code and AI code reviews as the same "AI" is not correct. The reviewer is using a fresh context window with no previous knowledge of the changes. That is why it's powerful, because you get the agent to interrogate the code without any preconception about the changes.
I found the Devin reviewer to be very good, and have heard good things about Cursor's Bugbot. I've also found asking an agent with fresh context or subagent to adversarially review locally is good.
The noise is a huge problem, indeed. Still, a panel of review agents using models and harnesses different from the one implementing a set of changes has proven immensely useful for myself. The panel is basically an n×m matrix of agents and highly specific review prompts, i.e.:
- review for intent fulfillment: is the ticket done?
- review for correctness: race condition bugs, ...
- review for security: check against this list of sources and best practices
- review for api conformity: identify all surfaces of systems outside this codebase touched by the code changes and check against their docs
- etc. pp., same for maintainability, observability & analytics, test coverage, usage of feature flags
The matrix is sparse, so not every model is used with each of the review categories. Effort levels vary, too. The next stage does a consolidation across all findings, then another stage spins up one agent per finding and investigates the whole codebases for identical / similar instances of the finding; finally, it suggests a fix.
This works extremely well for finding deficits, but the amount of noise drives me insane, too. Lots of feedback is technically correct and "by the book", but pretty useless in practical terms – or even detrimental because the amount of code written and thus the size of the change set explodes. I'm not yet sure how to tackle this problem, any suggestions are welcome!
Where I find it shines it to find inconsistencies. My readme or docs or ADR something should work like X but it finds a test where it tests something different and the test is green. Or other similar drift.
Yes, your prompt need to include to look for certain “quality” aspects you care about. But once that is there it can help find a lot of things.
It can also help in finding edge cases. It is really about the prompt.
What really important things are human reviews catching in your org?
I just feel more and more like the effort invested in manual reviews is not worth it
1. Does the implementation fit in the architecture/style of the project?
2. Are there potential security, accessibility, performance, etc. issues?
3. Domain specific knowledge (SQL, ASP.NET, XQuery, etc.) where there are better ways of solving a problem, or possible issues not handled.
4. Sense checking ... is the code easy to read? does it need an explanatory comment? does it need named parameters? etc.
1. Whether the thing should be done in the first place
2. If it's the correct solution on a high level
3. Whether it conflicts with or duplicates other parts of the system
4. Whether the comments are actually useful or restating the LLM chat
Also many others but these are the most common IME
All of these are angles an AI reviewer can test for as well, and will (IME) mostly catch mistakes correctly. I also still manually review code, and usually also catch issues, but the severity of what I find shrinks ever further as agents get better.
The sprawling code comments are becoming the most draining part of code review though, that's really killing me from the inside.
> All of these are angles an AI reviewer can test for as well, and will (IME) mostly catch mistakes correctly.
No, none of today's AI would give you enough signal around "should this thing be built in the first place" nor if it's the correct solution on a high level.
They don't understand why you are doing what you are doing, and even if you explain it, they still don't actually understand the motivation and lots of other things.
You'll get them to do guesses and pretend they actually know how to prioritize and will tell you it makes lots of sense, whatever they come up with. But try following it blindly and you'll see where you end up.
This is why "one agent + one good developer" beats "thousands of agents working in a swarm" still today.
The AI review are still quite far from having the same level of critical thinking and high level knowledge of your application, what you have done in the past and want to do next etc.
If you don't master this for your own project, what's even the point of your job.
Code review tools are designed for less organised dev teams that don’t do PRs and mandatory human reviews already.
It is papering over a lower level of competency without having to invest in actual human oversight or real process improvement.
That's a thoroughly uncharitable view. Especially in smaller orgs with a minimum velocity dictated by the company's need to survive, the amount of code required to be written just to keep up with your competitors is massive. Trying to review that all by hand, thoroughly, is draining, thankless, and tedious. You end up with a few fast movers producing most of the code, and some slower movers forced into a reviewer role they never signed up for. It's an unhealthy dynamic.