I recently asked it to give me code to do gapless playback of audio files using Apple's AVAudioEngine APIs. It got it wrong and additional prompts to explain why it was wrong didn't help.

To me what it seems like these tools do really well is paraphrase stuff that's in their training data.