what do you mean by reset at every turn? context stays until compaction. if you remove the reasoning tokens after every turn you will be constantly blowing cache which is far worse than filling up context.
Modern protocols loop back the reasoning tokens in raw textual form via an encrypted parameter. You can't see them (modulo the recent attack), but you do resubmit them.
There was an attack that convinced models to leak the contents of the reasoning tokens, and it came back as text that matched the length of the encrypted data very well. So it's probably still tokens.
Looping back latent state isn't that easy. The hidden data is the entire contents of the KV cache which can be massive. I don't think any provider is trying to loop the KV cache through the client, and neural compressions of the reasoning would be lossy / an advanced technique that is still firmly in the realm of research papers, as I understand.
Does it reset at every turn? From my experience in Codex for example, Luna (Max) fills the 256k token window relatively quick. The only thing lowering the context window again is the compaction.
Yeah, I remember reading about Anthropic's experiments with dropping out different things, and dropping the thinking is ok for compaction but pretty awful while it still fits in the context window. I could imagine doing it adaptively -- take a snapshot before the thinking, have a lesser model detect if there's a lot of spinning going on and reset to snapshot + result if so, otherwise accumulate. It'd also be interesting to have a lesser model rewrite thinking to be more streamlined (let's pretend the AI went directly down the right path on the first try). But it may be a disaster anyway -- isn't the thinking section something like a 1d best-logit slice of something like a 2d process, where the real processing could easily be happening under the visible surface?
I'm not sure if that's right. I only just recently learned that the "snapshots" (aka cached tokens) necessarily contain the entire context history, not just an image of a "state as of the final token" (well, it is the state as of the final token, but that state contains the whole history). So I'm not confident of my grasp of the structures here.
what do you mean by reset at every turn? context stays until compaction. if you remove the reasoning tokens after every turn you will be constantly blowing cache which is far worse than filling up context.
That's not my understanding of how most agents work. This is what a chain of request/response looks like:
So reasoning gets dropped from context and you still get cache from the accumulating requests.Edit:
I've realised I was incorrect, the thinking doesn't get passed back and forth but the latent snapshot does which result in using memory just the same.
Modern protocols loop back the reasoning tokens in raw textual form via an encrypted parameter. You can't see them (modulo the recent attack), but you do resubmit them.
Yeah I've done more research and that's what I meant in the edit you replied to.
But that's not the full reasoning token context, just a snapshot of the latent state at the end of it, no?
Have a look at the gemini ones they're pretty small.
There was an attack that convinced models to leak the contents of the reasoning tokens, and it came back as text that matched the length of the encrypted data very well. So it's probably still tokens.
Looping back latent state isn't that easy. The hidden data is the entire contents of the KV cache which can be massive. I don't think any provider is trying to loop the KV cache through the client, and neural compressions of the reasoning would be lossy / an advanced technique that is still firmly in the realm of research papers, as I understand.
Does it reset at every turn? From my experience in Codex for example, Luna (Max) fills the 256k token window relatively quick. The only thing lowering the context window again is the compaction.
I mean the thinking does not bloat the context window because it gets dropped at the next request.
It doesn't. It's called preserved reasoning and every recent reasoning model does it
Sorry, I've realised I was only partially correct.
Gemini[0] for example passes along a snapshot of the reasoning state but it's not the equivalent to keeping all the reasoning tokens in the context.
[0] https://ai.google.dev/gemini-api/docs/thinking#signatures
Edit: Apparently it does take the same space in the LLM latent space so I was wrong.
Yeah, I remember reading about Anthropic's experiments with dropping out different things, and dropping the thinking is ok for compaction but pretty awful while it still fits in the context window. I could imagine doing it adaptively -- take a snapshot before the thinking, have a lesser model detect if there's a lot of spinning going on and reset to snapshot + result if so, otherwise accumulate. It'd also be interesting to have a lesser model rewrite thinking to be more streamlined (let's pretend the AI went directly down the right path on the first try). But it may be a disaster anyway -- isn't the thinking section something like a 1d best-logit slice of something like a 2d process, where the real processing could easily be happening under the visible surface?
I'm not sure if that's right. I only just recently learned that the "snapshots" (aka cached tokens) necessarily contain the entire context history, not just an image of a "state as of the final token" (well, it is the state as of the final token, but that state contains the whole history). So I'm not confident of my grasp of the structures here.