apparently it got docked points for editing files out of scope
This must not be weighted very heavily on the benchmark because if it was, Opus would bomb every test (half kidding)
Do you have a source for this? That would explain it, but could be a bit of a concern on the general focus the model at higher thinking exhibits.
This must not be weighted very heavily on the benchmark because if it was, Opus would bomb every test (half kidding)
Do you have a source for this? That would explain it, but could be a bit of a concern on the general focus the model at higher thinking exhibits.