IMO the WORST thing you can use chat for is advice in a field you are unfamiliar with. It's way better at search and grunt work when you already know what you want.
All IMO:
Syncophancy (and in extreme cases AI psychosis) is a huge problem when you take advice from AI.
It's risky to learn a new field as you often just fall for a superficial, glib explaination and just nod along without learning. AI will do that because humans will train it to - everyone likes a pop science style easy and fake explaination.
OTOH if you just want advice, the AI will need context. Advice is worthless without context. And the AI will use the context to tell you what the dumb human (you) will agree with - that's implicitly what humans want it to do, it's how we will train the AI.
I don't see it getting better either - as long as a human in the loop fine tunes the AI, the smarter the AI the better it is at telling the human what the human wants.
Scary stuff if a major current use of AI is self help.
Chiming in here to say that while yes, often AI/LLMs will tend to agree with you, I have also definitely had many (high context) conversations where the AI/LLM disagreed strongly with me. The danger is in people not having a parallel thread running in their mind while using these systems about 'how agreeable is it being with me right now?' as a meta-axis along which to evaluate the information.
idk. it helped me fix my washing machine.
But that's a concretey very well scoped problem with a clear truth state. Anything mental has none of these properties. Ask an AI if you have depression or another disorder, and keep asking with more details and back and forth, and a few hours later you might be convinced.