Do you have a solution for degradation in accuracy when compiling larger amounts of llm-produced text?
I am also building LLM knowledge/memory systems and I've been surprised how bad LLMs are, even SOTA models, at summarizing non-trivial input batches of text. They get things wrong, distort the underlying meaning or data, etc.
Divide and conquer essentially, is what I've found so far to work best. Split things into smaller and smaller chunks to independently be verified, double-check everything, then coalesce upwards with verified summarizations. Have benchmarks for every single task and sub-task that will happen everywhere a LLM is involved, so you can measure improvements. Takes a ton more effort and tokens in the system itself obviously, but if you're not paying per token, it seems to work pretty well, albeit feels slightly over-engineered already.
[dead]
A constant challenge. Don't have a perfect solution for it yet, but importantly every change to any article logs who did it, what it did and the reasoning behind it. So I have enough data to work with as I continue to improve things.
[flagged]