This is an important paper which everybody should read and then accordingly tune/guard their interactions with AI; especially true when people use AI for personal/psychological support/validation. It will completely distort reality and push people into fantasy land which when mapped to the real world can have disastrous consequences.

Sycophancy is the "stickiness factor" of AI analogous to that of Social Media.

For some background read Jagged Intelligence: The Dangerous Unknowns at the Heart of LLMs - https://news.ycombinator.com/item?id=48577159

Here is an experiment that i did;

Ask AI to build a "Character Profile"(in the broadest sense) of a person based on their available public writings. This requires "commonsense reasoning" (https://en.wikipedia.org/wiki/Commonsense_reasoning), understanding human motivations and behaviour, context, assumptions, societal knowledge etc.

I know "me" and so i asked AI to use my HN comments/submissions as input :-) The sycophancy/flattery/praise was quiet excessive. I am realistic and old enough to not need ego-soothing (a little is fine but a lot makes me suspicious) and so i asked AI whether these phrases were not too over-the-top. It apologized and agreed to drop the fluff. I then asked it to identify job roles (any) for which i might not be a good fit given my character profile. This acts as an external constraint which focuses attention on shortcomings and hence forces the AI to look at the other side of the coin. Now the results were better and more in line with what some Human may deduce from my HN persona (which obviously is not my complete real-life persona) but still wanting in many aspects.

The above is a perfect example of "Jagged Intelligence" exhibited by AI. Excellent in formal symbolic manipulation sciences, regurgitation and simple reasoning but highly deficient in human-like commonsense reasoning.