Cached tokens count towards the limit as well. For example, if your context window is 50,000 tokens, it takes 9 requests to reach that limit without generating a single token.
Cached tokens count towards the limit as well. For example, if your context window is 50,000 tokens, it takes 9 requests to reach that limit without generating a single token.
Cached tokens counting toward the limit is ridiculous.
then it's basically useless lol, wtf, this has to be a defect
It's PR:
https://news.ycombinator.com/item?id=49556302