This is similar to how our college CS problem sets were graded. We were given a spec, and we had to implement a program that conformed to it. We had access to 70% of the test suite during development, and another 30% was hidden and only evaluated after submission. We were graded out of 100.
It was effective at making you think about the problem and anticipate what tests might be missing. I can see how this would be effective for coding agents, which tend to get progressively lazier at writing tests as session context grows.
The interesting twist here is that the test writer can also be wrong, the QA agent assigns blame when the implementation disagrees with a test case, and it’s not always going to be the implementation’s fault.
Makes me wonder if there should be problems for CS students where given a spec and an interface, but not an actual implementation, they write the tests instead.
Had this issue at uni back in '08 (physics), my code did what was asked of it, but the auto grader didn't like it. Nagged the professor until he had a closer look and fixed the issue, and asked me to resubmit.