Based on the numbers here it seems there’s less than 800 tasks in the entire benchmark. That is enough for a handful of engineers to comb through in a week (which is what OpenAI eventually did here).

On the one hand, kudos to them for actually doing that work.

On the other hand, garbage in, garbage out. It’s a bit embarrassing for the original authors to have not actually checked, and it’s embarrassing for everyone downstream to have not checked either.

Also if you check the article, although an LLM did find issues, it tended to underestimate issues that professional software engineers found.