IMO the unreasonable uselessness of LLMs is because for most tasks involving language the accuracy needs to be unbelievably high to have any real value at all.

We just don't have that.

We have autocomplete on steroids and many people are fooling themselves that if you just take more steroids you will get better and better results. The metaphor is perfect because if you take more and more steroids you get less and less results.

It is why in reality we have had almost no progress since April 2023 and chatGPT 4.