I asked Sonnet to port a prolog interpreter from Python to JS. I was surprised and delighted that it ended up writing a test suite for it. All tests passed.

Until... I inspected the code: it was just a bunch of print statements that said "test passed!" and didn't actually test anything.

That was last year, so hopefully it doesn't do that anymore.