That's neat and definitely the next step. But to be honest, I don't want an AI talk to me like that.

Same here.

Presumably it will be possible to adjust that behavior with settings, the system prompt, etc. Not that most users will make such adjustments, though.

I'm currently teaching a class on AI-related issues at a university in Tokyo. Many of the students were surprised when I showed them that they can change the response behavior of chatbots to make them more or less verbose, sycophantic, etc. It shifted the direction of our discussions on the possible impacts of AI on the people who use it.