> The problem is that the crackpots are out there writing, and this is snarfed into the training data.
No, the problem is the chatbot pretends to be an AI answer engine.
Recognise it is simply a zero-intelligence search engine, and you'll not be surprised at all to get crackpot writings from the web.
> write in a way that reintroduces rigidity
... and do not be surpised to get rigid crackpot writings from the web.
> Recognise it is simply a zero-intelligence search engine
I think that this highlights the problem: it's not even a proper search engine. If it were, it would find existing material and link to it. The chatbot instead re-elaborates the message. This is different enough to have legal consequences (see the recent German ruling).
By squashing everything together, one loses the ability to evaluate sources, including their credibility and ulterior motives. With a search engine, I can tell if I am getting climate information from the National Snow and Ice Data Center, from an oil trading consortium, or from a random blog.
The parent poster's suggestion is trying to work around the selection of sources, but such an attempt is fighting against the questionable design of the whole thing.
I don't think it's currently solvable, as selling those models as oracles is an essential component of their business plan.
I was using one as a way to search for recipes/ideas and I was really quite depressed when the thing replied with "I would recommend...".
No, you wouldn't because you're a fucking machine.
Plus this rot has extended to true search engines too. A recent Google blurb on new "AI" features of Google web search kicks of with the following misrep:
"The goal of Search has always been simple: to help you ask anything on your mind"
https://blog.google/products-and-platforms/products/search/s...
And they are also perverting search results, if this is to be believed:
"The Google AI Mode recognizes search intent by understanding the meaning of a query made rather than relying entirely on keywords. The platform uses sophisticated AI models to determine what users are looking to accomplish and whether they need any information, comparisons, or final decisions. This way, Google provides more accurate search results."
https://bostoninstituteofanalytics.org/blog/latest-google-ai...
Well, Sundar Pichai just left. In some "Why Google search got worse" threads, people who worked on search mentioned him championing predictable results without too much magic. Seems like it's mostly magic now.