> Qodo takes a different approach by starting with real, merged PRs
Merged PRs being considered good code?
What do you suggest they use for ground truth?
I thought about this quite a bit. There are some nuggets in the open source code:
- vX.X.1 releases. when software was considered perfect but author had to write a fast follow up fix. very real bugs with real fixes
- Reverts. I'm sure anyone doing AI code review pays attention to this already. This is a sign of bad changes, but as important.
- PRs that delete a lot of code. A good change is often deleting code and making things simpler
For the first, your signal would be weak, for those events are rare. I don't think deleting and reverting is a signal of quality. Rather, it demonstrates bad changes, as you said. This does not tell the model what good code is, just what it is not.
What do you suggest they use for ground truth?
I thought about this quite a bit. There are some nuggets in the open source code:
- vX.X.1 releases. when software was considered perfect but author had to write a fast follow up fix. very real bugs with real fixes
- Reverts. I'm sure anyone doing AI code review pays attention to this already. This is a sign of bad changes, but as important.
- PRs that delete a lot of code. A good change is often deleting code and making things simpler
For the first, your signal would be weak, for those events are rare. I don't think deleting and reverting is a signal of quality. Rather, it demonstrates bad changes, as you said. This does not tell the model what good code is, just what it is not.