>its information density is often really low
I find that it often isn't just too low, but also too high at the same time. It has a lot of information that is irrelevant, burying far too little relevant information for the work needed to extract it. I wonder if this is a side effect of attention mechanism, where the AI's ability to track information in a document is far less hampered by other information, leading to it organizing information to be both dense per word but any specific concept to be sparsely spread throughout the writing. I also wonder if this is why the order feels off, because the AI doesn't need for the information to be well ordered to the same extent that a human does, so the order of storing information is often overlooked when storing it into a document (and even information that needs to be ordered might have the ordering information separate from how it is ordered in the document).
IMO not enough talk about ordering. LLMs seem to have no understanding of e.g. the 5 Ws of journalism. Yes, they "know it", they just rarely use it. Nearly every programming response I get starts with the most lowest level fact "m_foo is [wall of text elided] hence the file format is wrong".
Whereas any human would say "I think your file format is missing a key/value pair for foo'ing. Let's trace through the code, and see if/where m_foo is set...."
It happens everywhere. Docstring type comments for a function endlessly just lists information about specific variables, implementation details, etc, and often never really even says what the function does. Then maybe a comment in the body might actually describe what it is for.
These days I've learned when facing these walls of text to check the last sentence first; it is often enough to provide the context for the text dump.
I have instructions to this effect in my claude.md. It doesn't help much on responses, but it give me a way to quickly say "reread claude.md and then rewrite that last prompt" and that can tend to go better.
Someone else mentioned model of mind while writing. The LLM has no model of mind; it does not consider how to present information, what you might already know or not know. We work in a huge mono-repo - it'll blithely just start listing variable names and I have no idea what file that is in, what it is referring to, how it interacts with 5 other classes it hasn't mentioned, or mentions out of order. It doesn't matter to the LLM. It can hold millions of facts in 'mind' at once; humans stall out at 5-7. If I replay saying it is leaving out info, many times it turned out it did dump all the facts out, spread across multiple prompts, often with key facts interspersed with all the vagueisms. No human mind can cope.