Either DeepSWE [0] or FrontierCode [1], depending on personal goals and requirements. The later is more interesting for me personally, due to the design of the benchmark heavily grading "mergability", i.e. how the provided output is to review and whether a serious developer can easily parse it and'd be willing to merge the result. In my mind and with my private evals, for quite some time I've held firm that a model can have a higher ceiling but that has limited value if I do not feel truly confident in signing off on the code.
Sadly no tasks for C, especially for working with optimized low-level data structures. And with testing performance of a solution. Anyone can write Python, try writing optimized low-level code.
Also I wonder if models playing dumb to prevent learning on outputs affected the score.
Also interesting that Claude edits files by writing and running Python scripts, is that efficient?
I just checked and for plain old C, there do not seem to be any reasonably comprehensive, current-day eval suites. Fully admitting that, even if there were, I couldn't assess their validity simply because I have never written or reviewed any C code in my life (something I should rectify probably). Maybe the closest proxy is just parsing through the experiences people claim to have whenever LLM assisted kernel development comes up [0], but if you have a dataset, experience, time and muse, I'd just go for it and do some tests yourself. Have been doing the same, mainly focused on code quality and dealing with a mix of Rust, frontend web tech and SQL which has been a small but meaningful project and part of my go to eval for over a year now.
I doubt that, in these tasks, model restrictions to prevent training are affecting the results, not least because for both evals, the labs provided pre-release model access and have an incentive to be seen as favorably. In any case, I have not seen regressions to prevent distillations myself even when working on microscopic model training projects with LLM assistance, what I have however reliably and consistently seen is that some providers do train on popular evals and can underperform with minor changes to the task due to that.
Yes, harnesses, including Claude Code can prompt the models to write throwaway code to execute certain tasks, mostly Python, bash scripts or TS/JS, with there being some biases towards one over the other depending on the lab or specific model. Mainly for repetitive tool calls with no pre-existing/provided tools enabling it. Is in most instances a lot more efficient then a model e.g. doing a refactor that requires consistent variable renaming directly and around Opus 4.1/GPT-5, models have been trained to very consistently and accurately gauge when a task can benefit from such scratchpad scripts vs when that is inefficient/not useful.
Also, GLM 5.2 seems to be the best open-weight model, and it beats proprietary Gemini and older versions of Claude which is amazing. You can have a model at level of Claude Sonnet 4.6 at home without sharing anything, and maybe even uncensor it.
Either DeepSWE [0] or FrontierCode [1], depending on personal goals and requirements. The later is more interesting for me personally, due to the design of the benchmark heavily grading "mergability", i.e. how the provided output is to review and whether a serious developer can easily parse it and'd be willing to merge the result. In my mind and with my private evals, for quite some time I've held firm that a model can have a higher ceiling but that has limited value if I do not feel truly confident in signing off on the code.
[0] https://deepswe.datacurve.ai/
[1] https://cognition.com/blog/frontier-code-1.1
Sadly no tasks for C, especially for working with optimized low-level data structures. And with testing performance of a solution. Anyone can write Python, try writing optimized low-level code.
Also I wonder if models playing dumb to prevent learning on outputs affected the score.
Also interesting that Claude edits files by writing and running Python scripts, is that efficient?
I just checked and for plain old C, there do not seem to be any reasonably comprehensive, current-day eval suites. Fully admitting that, even if there were, I couldn't assess their validity simply because I have never written or reviewed any C code in my life (something I should rectify probably). Maybe the closest proxy is just parsing through the experiences people claim to have whenever LLM assisted kernel development comes up [0], but if you have a dataset, experience, time and muse, I'd just go for it and do some tests yourself. Have been doing the same, mainly focused on code quality and dealing with a mix of Rust, frontend web tech and SQL which has been a small but meaningful project and part of my go to eval for over a year now.
I doubt that, in these tasks, model restrictions to prevent training are affecting the results, not least because for both evals, the labs provided pre-release model access and have an incentive to be seen as favorably. In any case, I have not seen regressions to prevent distillations myself even when working on microscopic model training projects with LLM assistance, what I have however reliably and consistently seen is that some providers do train on popular evals and can underperform with minor changes to the task due to that.
Yes, harnesses, including Claude Code can prompt the models to write throwaway code to execute certain tasks, mostly Python, bash scripts or TS/JS, with there being some biases towards one over the other depending on the lab or specific model. Mainly for repetitive tool calls with no pre-existing/provided tools enabling it. Is in most instances a lot more efficient then a model e.g. doing a refactor that requires consistent variable renaming directly and around Opus 4.1/GPT-5, models have been trained to very consistently and accurately gauge when a task can benefit from such scratchpad scripts vs when that is inefficient/not useful.
[0] https://news.ycombinator.com/item?id=44990981
Also, GLM 5.2 seems to be the best open-weight model, and it beats proprietary Gemini and older versions of Claude which is amazing. You can have a model at level of Claude Sonnet 4.6 at home without sharing anything, and maybe even uncensor it.
I've generally found DeepSWE[0] to be pretty true to reality.
[0]: https://deepswe.datacurve.ai/
Oh very interesting, I didn’t realize I should probably be using Fable Medium more than High, due to how that curve and the cost looks!
https://cognition.ai/blog/frontier-code (disclaimer - was on the team - but also we covered swebench pro/deepswe issues in here as well.)
1.1 seems a lot better than the original release, which was a bit hyperbolic. excited to see the team keep iterating.
FrontierBench
do they have a website? I have found only paper PDF and it seems more general than SWE
strawberry
Why is this a problem? Its like asking a person how many elder futhark runes are in the word strawberry.
Unless you want to tack on bpe enconding table to every llm context its pointless
Coding contains many subtasks analogous to counting letters in a word accurately.