I built exactly this using my own codebases. The setup isn’t that complex; split the git history to just before the change, sandbox the agent with everything they will need at that commit and lightly modify rules so they don’t go searching outside the box. Then they get the same prompt (usually the ticket that began the work) and are graded against the accepted PR.
The thing that takes the most time is finding the examples. In my real dev flow it’s rarely ticket -> PR -> merge, things bounce around a lot more. So, even though the stated goal is to get away from one shots, that is basically the environment you have to set or else test for specific other outcomes (e.g. agent stopped and raised a question when it realized x).
It takes time to do, but I would really recommend it. Now I can push new open models through the batteries and see how they line up to past ones in a few days (I run them locally, it’s slow). It moves my sense of x model is good at y and bad at z to from vibes to a better heuristic (these still run at temp 1, heuristic is the correct way to think about outcomes IMO). It grounds it in your actual code and problem space.
My takeaway from my testing: in Rails or front end codebases, most models I test are competent and with a human in the loop they would accomplish their goal of getting to a mergeable PR. They are not as good as Claude and since I pay subsidized rates via subscription Claude still gets first pass. They are very worthwhile to layer in as reviewers and catch many issues. My anxiety about a rug pull by the frontiers has been turned way down. I would have to adapt to a local only flow, but it wouldn’t be much adaptation and the opens can deliver in their current state.
Pretty cool to hear about your process - I've been thinking about this space a lot so have some context about pain points here.
I've found that building a benchmark is something that's easy-ish on the surface level, but has many layers of technical complexity the deep you go down the rabbit hole.
For example, a tangent on grading. Do you use test passing and/or LLM graders? If you're using an LLM, it's fairly easy to spin up a basic LLM grader. But - is that grader calibrated? Is it biasing towards same-model agents? How many results do you need to be above the noise floor? What is it even measuring, and is that aligned with what you care about? Does it have the right information to make those judgement calls?
I'm super curious how what you've been using this workflow for? Also happy to answer any questions / discuss further!
Don't you need to run each model several times and take some average since their outputs can change for a fixed prompt? How do you test for that?
Yes at temperature 1 it would need to be averaged. The mechanical scoring is hidden specs through and if they don’t pass that it’s not often a result of temp, but instruction following or verification (different flavor of instruction following given what they are told to do). What’s surprising is just how well they do on this dimension even with the variance. On my tasks anyway, maybe Rails work isn’t that complicated. It’s probably not that I am an amazing ticket writer.
I have run multiple with the same model through issues to see the output. What differs run to run on the same model is style, likelihood to find subtle bugs and chance to correctly guess author intent. I don’t score them on those things though, only note them.
To me, multiple vertical samples is more useful to the coder who isn’t benchmarking these as their job. The average is abstract to you if you’re not going to pointing them at a ticket more than once, but seeing their output over 10 different tickets will give you a window into model tendencies, strengths and weaknesses and help you understand how to write for them. Overall the exercise has made me a better issue writer.
How do you grade them against the accepted PR?
I have a similar calibration workflow, too [0]; but I am under no illusion that my personal benchmarks remain secret.
Once we send code & prompts to the providers, it can no longer be considered private.
[0] And came to the same conclusion as you did: Claudes were better but not by much: https://news.ycombinator.com/item?id=48654635
hey this seems really interesting - what prompted you to test multiple agents on your codebase?
Curiosity and anxiety.
I rebuilt my entire workflow around agents so the unease that the frontiers would change something (access, pricing, availability) and lock me out of that were high. Also why I spent way too much on hardware (at least that can be deducted). Now the whole stack could run in my house and I feel much better about the situation.
Once I got the testing going though it is worth it for its own pursuit. Building processes around the dev process and trying to get the best outcomes is at least as fun to me as actually delivering client code. For the first time in my tech career I feel like I’m in a place with no maps. No one has done my experiments yet. I have a custom quant of K3 at Q5 that lets me get 10 tok/s on a CPU inference box (admittedly you need a 72GB Blackwell also). As far as I can tell no one else has done this. It’s such an exciting time!
do you have some time to chat? janak@withspecific.com
[dead]