So, reduces cognitive engagement with the actual task at hand, and forces a huge attention share to hand-holding.
I don't think you two are disagreeing.
I have noticed this personally. It's a lot like the fatigue one gets from too long scrolling online. Engagement is shallower but not any less mentally exhausting than reading a book. You end up feeling more exhausted due to the involuntary attention-scattering.
It would be analogous to having to double check the IDE added the lines of code I actually typed. That’s not a great productivity boost, it’s a toy still in many ways.
> reduces cognitive engagement with the actual task at hand, and forces a huge attention share to hand-holding
You're in some senses managing an idiot savant, emphasis on the idiot part, except they're also a narcissist who will happily go out of scope if you let it.
If you have management experience, the analogy is immediately obvious. If you don't, I can see how having speed run learning it with a kid running around with dynamite would be taxing.
Text completer is still the most apt allegory. If you ask it to open a file and do something with it, often it wont open the file and just text complete the output. That failure mode never happens in any sort of human, it just happens because its a text completer.
Many people hate when you don't anthropomorphize LLM though, but that is the best way to understand how they can fail so spectacularly in ways we would never expect a human to fail.
Except many of us not having the problems you are having. I don't have LLMs "fail so spectacularly".
Interestingly, I have friends who aren't coders who use LLMs for various personal needs, and they run into the same kind of problems you are describing. 100% of the time, i've found that it's that they do not understand how to work with an LLM. Once i help them, they start getting better results. I do not have any need to anthropomorphize an LLM. I do however understand that I can use natural language to get quite complex and yes ACCURATE results from AI, IF i know what i'm doing and how to ask for it. It's just a tool, not a person.