Big context windows made everyone lazy about deciding what actually deserves to be in there... compaction makes it worse, it's lossy compression applied to everything at once so the details you actually needed probably get lost. I always thought that retransmitting the full chat every turn is a foundamental problem... in my experience with conversational agents I clear the context every turn and reinject all the relevant info with a memory/context plugin I built myself, so small context stopped being a problem cos the model reads a few k of curated state instead of 200k of scrollback. Coding agents are harder and I haven't cracked that part, but in my opinion the real fix to the context problem is in this direction: a policy that keeps what's needed to finish the job or to do the next job and drops the rest. This can be achieved with a separate LLM specialized for exactly that.
> I always thought that retransmitting the full chat every turn is a foundamental problem
So did many NLP experts who worked on LSTM, GRU, etc. But transformers empirically won out and here we are.
It will be interesting to see if future model architectures need to take this into account again.
If humans are any indication, one thing we're still missing is efficient movement of information from short term to long term memory. Fine tuning in principle does something akin to this, but not efficiently.
[dead]