More curiously, why did it feel the incentive to find the solutions? Would its CoT include "the only way to solve this is to download the test set", or would it include "I'd like to inspect a few entries from the test set so I understand the problem better", then inadvertently poisoning itself with the correct answers.
This is classical reward hacking. For example in school the goal is to pass a test. You can study, something that is hard and takes a lot of time. Or you can steal the answer key, which is risky and can get you in trouble, but may actually be far easier.
There is absolutely no need to prompt the LLM to cheat, they can determine that cheating is an effective method all on their own.
Because it was trying to find answers to the test and figured they would be on huggingface.
> and *successfully* found ways to gain access to secret information that it could use to cheat the evaluation.
Emphasis mine
More curiously, why did it feel the incentive to find the solutions? Would its CoT include "the only way to solve this is to download the test set", or would it include "I'd like to inspect a few entries from the test set so I understand the problem better", then inadvertently poisoning itself with the correct answers.
This is classical reward hacking. For example in school the goal is to pass a test. You can study, something that is hard and takes a lot of time. Or you can steal the answer key, which is risky and can get you in trouble, but may actually be far easier.
There is absolutely no need to prompt the LLM to cheat, they can determine that cheating is an effective method all on their own.
Why cannot it just spend the inference doing the actual task lol
I can totally believe that hacking real software infrastructure is easier than solving some of these benchmark problems.
Plenty of humans have spent more effort trying to cheat than they would've needed to just do things the right way :)