I think the first goal of writing is to talk to yourself. Without an external, unchanging scratchpad, most people cannot keep their thoughts truly straight. It can feel otherwise, but trying to write something about any tricky topic even a few pages in length can take a long time to be self-consistent. Clarifying it takes even longer, especially when you intend to share it. Writing as thinking is definitely a thing. I believe it’s the trickiest part of understanding how such interesting things pop out of a language model trained to predict the next token and rewarded with pleasing sequences. Since it is not thinking, it proves that certain sentences (most?) can be generated thoughtlessly.
I like the term I've seen someone use for LLMs: a "thought partner." Truly it's the duck programming duck I've always wanted.
I say that LLM's are a pacifier. LLM output is to conscious output as a silicone nipple is to a breast.
Sure, at some level, there are some people who at least some times, cannot pecieve a difference. That does not prove that there is no difference or that the difference doesn't matter.
Don't just focus on the model, half the turns in a session belong to the human, it's an instrument that depends on how well the user wields it. It does not need to produce the novel insights, just to support the user to have them.