Yes, how do we know Opus 4.8 hasn't been trained on the SWE-Bench examples?
With a squillion dollars at stake per bench point, someone will have figured out a plausibly deniable way to game these benchmarks.
Yes, how do we know Opus 4.8 hasn't been trained on the SWE-Bench examples?
With a squillion dollars at stake per bench point, someone will have figured out a plausibly deniable way to game these benchmarks.
Ehr, the SWE bench examples are particularly horrible as those are just publicly available historical PRs. So if the models are trained on GitHub data, it will be included.
So almost by design that particular benchmark is tainted, and benchmarks recall rather than reasoning.
Wow that's worse than I thought, and breaks the number one rule of machine learning: you don't train the model with your test dataset.