Putting on my machine learning PhD student hat, the way to do this was to leave 10% of the tests out as a ‘test’ set and then once the port was done, bring them back in and find out how good the port is. The port may genuinely be good but because they spent 100k of compute hill climbing the whole test suite, “the test suite passes” now provides far less evidence that the port is good. Its weird that at Anthropic, a very ML phd company, no one pointed this out.
Huh? This approach makes sense for non-deterministic problems. Not engineering problems that have deterministic end results.
I’d want to estimate P(a random test passes) where the existing suite of tests is taken to be sampled from a distribution.
> non-deterministic problems
I'm guessing you mean probabilistic? Nevertheless, you have an indeterministic variable here which is what the LLM generates.