Trying something like this in a reusable way at https://github.com/mateffy/gesetz

It only tries to solve the „quality“ aspect of that equation tho, at least for the code output.

But I think for actual evals, some heuristics about a coding agent session are needed. But since the input tasks are always different, it’s hard to make any KPI actually comparable.

Did the agent needing 3 rounds of feedback on a big task perform worse than the one that needed none, but only worked on a small change?