Agreed - "underspecified prompts" being listed as a failure of the tooling is not a strong case. Even interns can understand ambiguous asks with a bit of help, and understand when they need to stop and ask instead of just carrying on. They are often working fairly independently on ambiguous tasks before the end of an internship, too.
So is the argument that frontier models are not just junior engineers, but first-month interns with no capability of progressing beyond that level?
> Even interns can understand ambiguous asks with a bit of help
This is not a case of an ambiguous task. This is literally trying to judge a model based on information it cannot possibly know, like trying to judge someone based on whether they know what I have hidden in my backpack. In the real world an intern could look at unit tests or ask for feedback, but that is not the case in a benchmark.
[dead]