The obvious limitation to this approach is that you're limited to doing evals for things where the SOTA models are capable of judging an attempt. Many tasks are not yet amenable to this, especially if you don't have a ground truth to have the SOTA model compare to.

And even for the tasks that are amenable to having LLM judges assess them, there's still a huge benefit in looking at traces yourself and labeling them. It's time-consuming, yes, but you'll learn a lot about the ways an agent fails in your particular domain, it gives you more reliable golden datasets, you have a mechanism to evaluate your judges, etc.

I haven't done this for coding yet (honestly can't really figure out the best approach), but for in-app evals, I built a whole interface to review traces. My app sends up 10k - 30k input tokens on the initial request, between fixed prompt, schema, and dynamic context, so just reviewing it is a huge pain. My interface converts all that raw input data to human-readable objects, highlighting important things and tying together inputs vs outputs for the request. Makes reviewing it, grading it, tagging it, etc. much easier. And then once I've done that for a batch of requests, the passing requests have their inputs and outputs frozen as a golden dataset that I can run other models against. And there the LLM judges are great, because they're very adept at looking at an input prompt, a given model response, and a (human-approved) ground truth, and pointing out any significant differences between the output and the ground truth. Much easier than actually producing the ground truth.

I highly recommend Hamel Husain's writings on evals.