You've overstated the conclusion. The SWE-bench series has had issues since its inception.
OpenAI no longer recommends SWE-Bench-Pro as a benchmark: https://openai.com/index/separating-signal-from-noise-coding...
You've overstated the conclusion. The SWE-bench series has had issues since its inception.
OpenAI no longer recommends SWE-Bench-Pro as a benchmark: https://openai.com/index/separating-signal-from-noise-coding...
That's smart, they only recommend benchmarks that make them look better then their competitors.