We did some measurement at the beginning of the year and found that our PRs were getting bigger and review time was coming down even without us doing anything, which is not a good sign.

We ended up moving peer review to the implementation plan rather than the PR, then having a loop that validates the code against the plan when the PR is raised. That way the agent gets a CI failure if it deviates from the plan, which it then fixes or acknowledges. Anything with no differences gets merged without human review, differences get approved by the original person who peer reviewed the plan.

I puled some stats the other day for a presentation I'm working on about what we did:

Matched plan on first pass: 25% (169/677) Had differences: 75% (508/677)

Differences per PR: Median 2 Mean 2.60 P90 6 Max 20

1,757 findings:

- Missing (skipped planned work): 44.6% of findings, 51.7% of PRs - Changed (done differently): 42.7% of findings, 51.3% of PRs - Beyond (extra, still in scope): 9.1% of findings, 19.8% of PRs - Scope (unplanned feature): 3.6% of findings, 6.6% of PRs