It would be interesting to try to control for system prompt although obviously there's some coupling to what the harness provides and the instructions the model receives on using it
It would be interesting to try to control for system prompt although obviously there's some coupling to what the harness provides and the instructions the model receives on using it
We'd rather not mess with system prompts, we just eval them as shipped to keep the results reproducible.