Was expecting the article to go more in-depth.
Say, what happens when chain of summaries grows so long, that it still overflows context window. Is summarization runned over the summaries in the context window?
Was expecting the article to go more in-depth.
Say, what happens when chain of summaries grows so long, that it still overflows context window. Is summarization runned over the summaries in the context window?
There is no chain, just a lossy summary that includes a summary of any previous summary along with the fresh messages.
What more depth is there to go to? Compaction is a single LLM call (practically) which can have some deterministic diffing/extraction baked in, or multiple LLM calls (generally wasteful). There's only 1 summary in the context window at one time. Every prompt goes [CONVERSATION_HISTORY] + input -> model turn. As soon as total context exceeds that it compacts, so there's no summarisation overflow (you can enable an agent to access past summarisations from past compactions, but the snake starts to eat it's own tail).
I think there are caveats anyway.
1. What happens if it overflows during assistant's turn, while it makes tool calls? Is it better to make a compaction in the middle of the chain of tool calls, or maybe before a potentially long chain of tool calls? If latter, how to choose the right point for compaction?
2. There should be enough space in the context window for a summary. What if, theoretically, a single new user message already overflows the context window? Or what happens if a summary is too long?
Maybe those are stupid questions, but I'm making a point that there is a room for going in-depth.