Your example sounds like a different case than what OP outlines since you know enough to reach the conclusion in the end.

An extreme example, but what happens if Bob Nottechnical follows GPT instructions which happen to include `sudo rm -rf /`? How would Bob know to stop trusting the instructions when every time he's followed things before it worked as expected?