I have an alternative view which is that sycophancy erodes trust in AI, from those who are not seeking validation from a machine, but information or advice.
If you're looking for advice in a situation where you are not sure of what the correct choices are, you will find LLM chat AI to go in circles. It says one thing. Something is dodgy about it, so you raise a tentative objection (as a non-expert). The thing does a "you are completely right, I apologize" about face and then says something different, and things have begun to slide into uncertainty.
That might not exactly be sycophancy, but it's basically the same thing: producing responses that are reflection of what is in the chat, rather than any real shit.
Pick any topic where people disagree. It could be an entirely technical topic in which engineers have settled the questions, and the only contrarians are crackpots. The problem is that the crackpots are out there writing, and this is snarfed into the training data. Crackpots use certain ways of talking about certain subjects. If you use similar vocabulary and concepts that align with some crackpot theory, the AI simply starts predicting tokens according to that, and you are now in crackpot land: what you are saying is validated using the crackpot terms. Next, write in a way that reintroduces rigidity: proper terminology and correct concepts, and, whoa, the AI is an engineer again, contradicting the previous crackpot shit.
When I want advice, I’ve found it useful to tell the model that someone else came up with the ideas we’re discussing, not me. That pushes the model away from sycophancy, and better calls out the risks. Sometimes it goes too far in shutting down the idea no matter what, but I think it’s easier to critically evaluate critiques than to step back from the model praising everything you say
I realize this is not scalable, but I’ve come to the personal preference that sycophancy be overt. LLMs are always steering in a direction and I find it a good reminder that they are just as incapable of objectivity as humans. TBD, but I think I’d prefer that fact plainly visible where I can see it.
This rings far truer for me. I don’t understand the psychosis of somehow allowing LLM responses to have an emotional impact on you, to make you think or believe a certain thing, to the extent that you blow up your own life over it - thank god I don’t have whatever flaw others have that make them ridiculously vulnerable to that.
One key point is that - “pick any topic where people disagree” - most knowledge-seeking I perform via a chat bot is to get a sense of what most people generally agree upon. A quick poll of the zeitgeist of the corpus it was trained on. What used to take me 10-15 minutes on Google now takes me 2-3 minutes with Gemini.
That holds true for Claude code too - I’m relying upon the fact that other people have already solved my problem, or have at least solved the components I need to tie together into a solution for my problem - I’m just fast forwarding through the process of digging them up myself and making sure they’re interoperable. The lion’s share of the tasks I set forth for an LLM is just ‘go find me something like this.’
How some people get from that, to weeks of AI-triggered mania, I simply do not understand.
> The problem is that the crackpots are out there writing, and this is snarfed into the training data.
No, the problem is the chatbot pretends to be an AI answer engine.
Recognise it is simply a zero-intelligence search engine, and you'll not be surprised at all to get crackpot writings from the web.
> write in a way that reintroduces rigidity
... and do not be surpised to get rigid crackpot writings from the web.
> Recognise it is simply a zero-intelligence search engine
I think that this highlights the problem: it's not even a proper search engine. If it were, it would find existing material and link to it. The chatbot instead re-elaborates the message. This is different enough to have legal consequences (see the recent German ruling).
By squashing everything together, one loses the ability to evaluate sources, including their credibility and ulterior motives. With a search engine, I can tell if I am getting climate information from the National Snow and Ice Data Center, from an oil trading consortium, or from a random blog.
The parent poster's suggestion is trying to work around the selection of sources, but such an attempt is fighting against the questionable design of the whole thing.
I don't think it's currently solvable, as selling those models as oracles is an essential component of their business plan.
I was using one as a way to search for recipes/ideas and I was really quite depressed when the thing replied with "I would recommend...".
No, you wouldn't because you're a fucking machine.
Plus this rot has extended to true search engines too. A recent Google blurb on new "AI" features of Google web search kicks of with the following misrep:
"The goal of Search has always been simple: to help you ask anything on your mind"
https://blog.google/products-and-platforms/products/search/s...
And they are also perverting search results, if this is to be believed:
"The Google AI Mode recognizes search intent by understanding the meaning of a query made rather than relying entirely on keywords. The platform uses sophisticated AI models to determine what users are looking to accomplish and whether they need any information, comparisons, or final decisions. This way, Google provides more accurate search results."
https://bostoninstituteofanalytics.org/blog/latest-google-ai...
Well, Sundar Pichai just left. In some "Why Google search got worse" threads, people who worked on search mentioned him championing predictable results without too much magic. Seems like it's mostly magic now.
Perhaps only a small minority want quality out of information product, while most want satisfying simplicity. This is definitely how the news plays out.
I recently built an ai chart product that attempts to add more nuance by injecting extra models to jump in when the first model misses the mark. (http://pellmell.au So far it seems like only a minority of people like it. The main feedback is that answers are confusing or too much to read
I learnt a long time ago that most people don't want their problems solved, they just want to vent.
You say that like its entirely a bad thing.
"They want to vent" :-
"they want to be heard" :-
"they want to be understood" :-
"they want to know they are understood" :-
"they want to feel they matter to and are safe with the people they value"