As both a writer and SWE the problem with LLM output is the same. They can achieve localized goals but have trouble juggling multiple concerns at the same time.
Things that are easy for humans, like remembering a character is holding a gun during a conversation, gets forgotten, because there is no clear architectural/spacial understanding of how the words are describing an underlying reality. And if the LLM gets this right it might instead forget which character has a Southern drawl, or make an illogical metaphor.
This is all more clear with programming, but the problem is the same - a lack of interwoven semantic understanding and friction between the real world and the self-contained world of the LLM's context.
Speaking of remembering, i just asked Astra about what temp to use and when to put on a pork shoulder for BBQ. It told me a temp and time and then went through and then did long winded explanation of all the variables that go into it's assessment and ended "in conclusion I recommendation X" where X was a totally different conclusion that in had earlier.
It also insisted that I start an hour early even though it was adding in 3 hours of rest time at the end.
Human judgment it is not
Chat: https://chatgpt.com/share/6add218a-3bc8-83ea-a2d7-613444eb7c...
No. The conclusion was not totally different. It recommended one of the explained options and recommended one extra hour as a buffer. I feel like this is a distinct reading comprehension issue if you think the final recommendation came out of left field.
Yeah but that entire wall of text was not needed. The user wants a simple answer that they can take action on.
This happens to me way too often. I just want a simple answer. I don't need a wall of text with 6 different options and a lengthy explanation. Just. Tell. Me. The. Answer.