So far this year, of the defects severe enough to warrant a formal RCA process that I’ve dealt with, “AI generated tests asserted that the defective behavior was correct,” was a contributing factor in every single one.
So far this year, of the defects severe enough to warrant a formal RCA process that I’ve dealt with, “AI generated tests asserted that the defective behavior was correct,” was a contributing factor in every single one.
I haven’t had this happen yet, but I was just thinking about this last night. I would be very interested to see if exercising TDD, as difficult as that practice is to actually do, would have a meaningful positive impact on improving this phenomenon. Usually agents model what most humans do, writing tests to pass and assert that the code you just wrote works as you wrote it.
I’ve had decent luck with it as long as I at least stub out the test cases with a test name and a “given … when … then” comment.
Otherwise it seems the AI has a tendency to write the worst kind of TDD antipattern microtests that simultaneously make refactoring impossible while failing to document or even confirm the most important functional requirements.
Once it’s got those stubs it does do a pretty good job of suggesting behaviors I forgot to specify, and other improvements. It’s just “designing” test fixtures from the ground up that seems to put it into “generate plausible-looking slop” mode.