I think it should, as you dont need to use the encoder layer on the new tokens, you just read the embedding from the cache. that's why cache reads are cheaper
It's supposed to be for token optimization (https://code.claude.com/docs/en/prompt-caching), but are people experiencing degraded performance when you let Claude Code sit for hours/days and come back?
The cache shouldn't affect inference. It is purely an I/O optimization.
I think it should, as you dont need to use the encoder layer on the new tokens, you just read the embedding from the cache. that's why cache reads are cheaper
I meant, it shouldn't affect the resulting LLM output. It's a performance optimization that doesn't change the behavior.
Is that from start of a new conversation per conversation?
It's supposed to be for token optimization (https://code.claude.com/docs/en/prompt-caching), but are people experiencing degraded performance when you let Claude Code sit for hours/days and come back?
yes, 100%.