Building ad-hoc evals is trivial these days. And you can place an LLM judge in front to disambiguate between output. e.g. I have Frigate gating a video feed so that an LLM can watch our home cameras and the labeling task takes a few minutes and then the evals run at fairly low cost https://wiki.roshangeorge.dev/w/images/4/4e/Screenshot_-_Eva...
This means that generic benchmarks and evals are sort of passé. There's no need to have them write "make a coffee shop page" or whatever. Instead, simply focus on the real problem you have and evaluate whether they solve that problem. You want to move along the price-performance frontier alone and the SOTA models are decidedly at the top of the performance curve but very expensive so they serve very well to produce the gold standard and to judge.
The change from prior to today is that benchmaxxing and per-token pricing allowing for evaluation point to the same direction: do not proxy results. Instead deploy the highest end model you have as a judge for others until you have statistical confidence in discrimination and move along the task-specific frontier.
The obvious limitation to this approach is that you're limited to doing evals for things where the SOTA models are capable of judging an attempt. Many tasks are not yet amenable to this, especially if you don't have a ground truth to have the SOTA model compare to.
And even for the tasks that are amenable to having LLM judges assess them, there's still a huge benefit in looking at traces yourself and labeling them. It's time-consuming, yes, but you'll learn a lot about the ways an agent fails in your particular domain, it gives you more reliable golden datasets, you have a mechanism to evaluate your judges, etc.
I haven't done this for coding yet (honestly can't really figure out the best approach), but for in-app evals, I built a whole interface to review traces. My app sends up 10k - 30k input tokens on the initial request, between fixed prompt, schema, and dynamic context, so just reviewing it is a huge pain. My interface converts all that raw input data to human-readable objects, highlighting important things and tying together inputs vs outputs for the request. Makes reviewing it, grading it, tagging it, etc. much easier. And then once I've done that for a batch of requests, the passing requests have their inputs and outputs frozen as a golden dataset that I can run other models against. And there the LLM judges are great, because they're very adept at looking at an input prompt, a given model response, and a (human-approved) ground truth, and pointing out any significant differences between the output and the ground truth. Much easier than actually producing the ground truth.
I highly recommend Hamel Husain's writings on evals.
> Building ad-hoc evals is trivial these days.
I highly doubt that you have solved it. Writing proper evals and benchmarks for real-world scenarios is far from trivial.
Benchmarking an agent essentially means freezing, at the very minimum:
- the model
- the model's configuration (e.g. effort, permissions, provider)
- the dataset (e.g. a git repository at a specific sha)
- the code running the agent itself (you can build your own harness, trivial, but you still need to ship it as a single executable, froze in time. benchmarking against closed source runtime like claude code is quite useless, they change too frequently and in ways you cannot directly inspect).
- the tools at agent's disposal. Even a slightly different implementation of tool X (e.g. grep or readfile or sed) has an impact. In general this implies also freezing a very specific container image. In my personal benchmarks I provide a specific list of tools that come with the executable, there's no possibility of interacting with the outside world besides the provided apis, the agent bundles its own tools.
And even then: there's significant noise coming from the LLM providers themselves which noticeably change the models behaviour, I don't know whether this is because they optimize some settings or change the inference over time, etc.
And, last but not least, the output of LLMs is non deterministic.
Also, the LLM as judge presents essentially the same non-deterministic problems, has to be benchmarked itself thoroughly, and writing quality rubrics or "golden answers/outputs" is just difficult. One of the metrics I consistently try to emphasize is to avoid the "shotgun vomit dump" of information. So answers that get right to the point in plain terms avoiding dumps of information filled of jargon on top of the user are rated differently.
In short: its far from trivial to benchmark models on real-world agentic work taken from your personal or professional projects.
And even creating the test cases themselves is hard. No: you cannot take the output of some "sota" and use it as gold standard. This is a very crap approach. It's the sloppiest solution to the problem, in the very sense of slop: plausible, average, lacking any creativity or out of the box thinking, the things that make the real difference in complex software development.
The very point of creating these benchmarks is to find which configuration/model/tools/harness (skills/mcps/documentation/agents.md, etc) works better.
And it only works if you create these benchmarks yourself from genuinely difficult non-trivial work and find a solution that is better by most metrics implementation-wise, albeit you could settle on the implementation solving a series of cases and edge cases.