> I don't think that follows. This is just LLMs being, for a lack of a better word, "gullible." How is it different from a person believing whatever they read on the Internet? People fall for spam and scams all the time, doesn't mean they are just glorified searches ;-)

To my knowledge, AI still can’t tell the difference between correct and incorrect. It can’t tell if something doesn’t make sense. That’s why to this day AI still makes mistakes on simple things.