> If it's knowledge (the intricacies of some arcane OS API, for example), an LLM can do very well
Only if that knowledge is sufficiently represented in the training data or on the web. If, on the other hand, it’s knowledge that isn’t well (or at all) represented, and instead requires experience or experimentation with the relevant system, LLMs don’t do very well. I regularly fail with applying LLMs to tasks that turn out to require such “hidden” knowledge.
> If, on the other hand, it’s knowledge that isn’t well (or at all) represented, and instead requires experience or experimentation with the relevant system, LLMs don’t do very well. I regularly fail with applying LLMs to tasks that turn out to require such “hidden” knowledge.
It's true enough that there are many tasks like this. But there are also many relatively arcane APIs/protocols/domains that LLMs do a surprisingly good job with. I tend to think it's worth checking which bucket a task falls into before spending hours or days hammering something out myself.
I think many devs are underestimating how arcane the knowledge needs to be before an LLM will be hopeless at a knowledge-based task. There's a lot of code on the internet.
And if it's really well represented, then it's hopefully already in a superior library or documentation/guide, and the LLM is acting as an (untrustworthy) middleman.
If the code can be generated correctly, is it controversial to say that generating it will be more efficient than reading through documentation and/or learning how to use a new library?
If you grant that, the next question is how high the accuracy has to be before it's quicker than doing the research and writing the code yourself. If it's 100%, then it's clearly better, since doing the research and implementation oneself generally takes an hour or so in the best scenario (this can expand to multiple hours or days depending on the task). If it's 99%, it's still probably (much) better, since it will be faster to fix the minor issues than to implement from scratch. If it's 90%, 80%, 70% it becomes a more interesting question.
Compare to: "If you can copy-paste from a Stack-overflow answer, is it controversial to say that copy-pasting is more efficient than reading through documentation and/or learning how to use a new library?"
If I understand the code and it does exactly what I need, should I type the whole thing out rather than copy-pasting? Sounds like a waste of time to me.