> tool call pruning breaks cache and people will tell you this is horrible and expensive
> except i looked at some anthropic data and real user behavior ends up with better cache hits and 30% less spend
> even this is needs to be analyzed further, it's just not simple
> for openai data it's inverted! cache hit ratio is actually better [sic: I think he meant worse based on the screenshot] with tool call pruning turned on
> but the net $ saved is only 5%
> kimi is a funny one - it has better cache hits with pruning on...but is also more expensive!
There was also another thread recently where he discussed that pruning improves user experience (models are smarter with less context) but I can't find it.
My understanding of caching with most models/providers is that a prefix substring of the context has to be reused for a cache hit, but not necessarily the whole entire context window. So if you prune tool calls from the history, you're going to get one cache miss on the newly-pruned history, and then you're going to be getting cache hits on every subsequent turn, with a lower number of input tokens. If you prune subsequent tool calls after that, you would still get a cache hit for the already-pruned portion of the context, just not the full context.
The way you do this (and the way opencode does it) is you do most of your pruning in more recent history. Last I looked at opencode, they start pruning tool call results after 2 full agentic turns. So you probably dont get quite as good hits on cache for the most recent 1-5% of your turns, but after that everything else caches fine and those tool calls that likely aren't relavent to your session anymore are gone.
> our implementation is it only prunes calls from > 3 user messages ago, if context is > 40K, and only if there's at least 20K tokens to be removed
Seems reasonable to me and explains why I can have long sessions (way longer than with zed agents) while still hitting cache. Opencode is just missing per-provider TTL.
I found that keeping current context utilization at 18% of total context length was best for minimizing spend, across all models with 400k context length or more
They are. Empirical evidence on my side. Because attention is sparse across the context. It's not truly treating a million token the way it treats a fraction of that count. For performance.
I can't confirm this. Having utilized Opencode for a large project over the past 10 months, with multiple models and agents, we've never run into such 'cache stability issues'."
> Ah, reminds me of good old "There are only 2 hard problems in computer science: cache invalidation, naming things, and off-by-1 errors."
You quip, but LLM KV caching (from the harness side) is quite easy: You get a cache hit on stable prompt prefixes, period. That means you want to keep the prefix stable, and only append at the end of the conversation.
Made up example: Don't put the git branch name into the system prompt part (that comes first), as whenever the branch name changes, that'd trigger a cache invalidation of the entire prompt.
Getting this right requires some care to not by accident modify the prefix, basically, and some design on communicating the things that can change (user configuration, working dir, git information, ...).
That sounds like the experience of writing Containerfiles; since steps are cached you want to pull the thing you are iterating on as far down as possible.
All of this work has been done before in different contexts. Memory management with bigger blocks and weaker definitions that change whenever some grad student gets a bright idea.
100%. Since you mention memory management: Generational GC is pretty much the same idea: Keep the stuff that's least likely to change an important property (liveness) together.
Conceptually the underlying general idea is to sort things based on stability if you can avoid recomputing properties of the stable part.
I'm not sure that is really the case, or relevant in practice. I have been using OpenCode with DeepSeek lately (regular coding). For instance, today I got 120 million input tokens hitting cache, vs just 2.59million missing cache.
Reads like a LOT of tokens to me. What does your usage /workflow look like? I'm v curious because although I do use Claude code, my token counts aren't nearly as much
Not OP, but I routinely load 150k tokens into context. A full sub-package to work on, select other files in the monorepo, e.g. front-end visualization and back-end data loader. Then work some 150k tokens, then start again.
At the end, cache hit rate is like 99.5% if Novita is not having issues.
For official DeepSeek API, 99.9% or something.
Custom harness that never compacts or otherwise doctors the history.
What I noticed when using OpenCode with llama.cpp, was that the default host RAM prompt cache size in llama.cpp was way too small for say 128k Qwen3.6 27B.
The default is just 8GB and a full 128k context for the dense model can take most of that. So then comes an agent and causes eviction and subsequent cache miss.
Bumped the cache size (--cram IIRC) up to 48GB and had much better results.
Here are my stats (from DeepSeek directly, with a script I wrote). The prices are what equivalent Sonnet usage would have cost, the actual amount I paid was $10. On performance, DeepSeek V4 Pro is comparable to Sonnet for me.
The OpenCode devs talk about this on Twitter a lot, e.g. https://xcancel.com/thdxr/status/2048268697790300343
> tool call pruning breaks cache and people will tell you this is horrible and expensive
> except i looked at some anthropic data and real user behavior ends up with better cache hits and 30% less spend
> even this is needs to be analyzed further, it's just not simple
> for openai data it's inverted! cache hit ratio is actually better [sic: I think he meant worse based on the screenshot] with tool call pruning turned on
> but the net $ saved is only 5%
> kimi is a funny one - it has better cache hits with pruning on...but is also more expensive!
There was also another thread recently where he discussed that pruning improves user experience (models are smarter with less context) but I can't find it.
This can also be disabled in the config: https://opencode.ai/docs/config/#compaction
My understanding of caching with most models/providers is that a prefix substring of the context has to be reused for a cache hit, but not necessarily the whole entire context window. So if you prune tool calls from the history, you're going to get one cache miss on the newly-pruned history, and then you're going to be getting cache hits on every subsequent turn, with a lower number of input tokens. If you prune subsequent tool calls after that, you would still get a cache hit for the already-pruned portion of the context, just not the full context.
So it makes sense to first send stable prompt, reasoning and files content, tool calls summary and actual tool calls at the very end?
The way you do this (and the way opencode does it) is you do most of your pruning in more recent history. Last I looked at opencode, they start pruning tool call results after 2 full agentic turns. So you probably dont get quite as good hits on cache for the most recent 1-5% of your turns, but after that everything else caches fine and those tool calls that likely aren't relavent to your session anymore are gone.
You didn't quote the interesting part:
> our implementation is it only prunes calls from > 3 user messages ago, if context is > 40K, and only if there's at least 20K tokens to be removed
Seems reasonable to me and explains why I can have long sessions (way longer than with zed agents) while still hitting cache. Opencode is just missing per-provider TTL.
I found that keeping current context utilization at 18% of total context length was best for minimizing spend, across all models with 400k context length or more
They are. Empirical evidence on my side. Because attention is sparse across the context. It's not truly treating a million token the way it treats a fraction of that count. For performance.
I can't confirm this. Having utilized Opencode for a large project over the past 10 months, with multiple models and agents, we've never run into such 'cache stability issues'."
That'd be really easy to spot and also fix, most likely. Any open issue you could point us to, must surely been reported already?
> That'd be really easy to spot and also fix, most likely
Ah, reminds me of good old "There are only 2 hard problems in computer science: cache invalidation, naming things, and off-by-1 errors."
> Ah, reminds me of good old "There are only 2 hard problems in computer science: cache invalidation, naming things, and off-by-1 errors."
You quip, but LLM KV caching (from the harness side) is quite easy: You get a cache hit on stable prompt prefixes, period. That means you want to keep the prefix stable, and only append at the end of the conversation. Made up example: Don't put the git branch name into the system prompt part (that comes first), as whenever the branch name changes, that'd trigger a cache invalidation of the entire prompt.
Getting this right requires some care to not by accident modify the prefix, basically, and some design on communicating the things that can change (user configuration, working dir, git information, ...).
That sounds like the experience of writing Containerfiles; since steps are cached you want to pull the thing you are iterating on as far down as possible.
All of this work has been done before in different contexts. Memory management with bigger blocks and weaker definitions that change whenever some grad student gets a bright idea.
100%. Since you mention memory management: Generational GC is pretty much the same idea: Keep the stuff that's least likely to change an important property (liveness) together.
Conceptually the underlying general idea is to sort things based on stability if you can avoid recomputing properties of the stable part.
It's even closer to prefix matching on super long strings by chunk
[dead]
Opencode (and other coding agents) have hundreds of open issues reported. It is quite discouraging when they are not being closed/fixed.
These projects have also been the recipients of PR spam, lots of duplicates and unconfirmed in there for less technical people and clawd operators
[dead]
https://github.com/anomalyco/opencode/pull/14743
I'm not sure that is really the case, or relevant in practice. I have been using OpenCode with DeepSeek lately (regular coding). For instance, today I got 120 million input tokens hitting cache, vs just 2.59million missing cache.
Reads like a LOT of tokens to me. What does your usage /workflow look like? I'm v curious because although I do use Claude code, my token counts aren't nearly as much
I want to know if I'm missing something cool!
Not OP, but I routinely load 150k tokens into context. A full sub-package to work on, select other files in the monorepo, e.g. front-end visualization and back-end data loader. Then work some 150k tokens, then start again.
At the end, cache hit rate is like 99.5% if Novita is not having issues.
For official DeepSeek API, 99.9% or something.
Custom harness that never compacts or otherwise doctors the history.
Those numbers make sense to me...120 million input tokens is like 120 sessions of hitting the full context limit, which seems like a lot to me though
What I noticed when using OpenCode with llama.cpp, was that the default host RAM prompt cache size in llama.cpp was way too small for say 128k Qwen3.6 27B.
The default is just 8GB and a full 128k context for the dense model can take most of that. So then comes an agent and causes eviction and subsequent cache miss.
Bumped the cache size (--cram IIRC) up to 48GB and had much better results.
I am getting 98.6% cache hit ratio on deepseek-v4-flash with opencode
That’s impressive!
On the sheer performance it’s comparable to Opus ?
Here are my stats (from DeepSeek directly, with a script I wrote). The prices are what equivalent Sonnet usage would have cost, the actual amount I paid was $10. On performance, DeepSeek V4 Pro is comparable to Sonnet for me.
All of this usage was with an OpenCode subagent exclusively.[flagged]
out of curiosity, how do you measure cache hit rate in opencode ?
opencode stats
So the calculation is:
Total input token = input + cache read + cache write Cache hit rate = cache read / total input token.
That is 71% in my very limited use of opencode.
The first
There are some that are specific to certain models like qwen/gemma
I switched to vLLM and those went away. Need to look at my opencode config and adjust some others based on things I see here
[flagged]