The automated version of this is mutation testing.

Which is actually probably a solid idea for this exact use case.