Not sure if this will help you but it helped me. I was using subagents and workflows for lots of long running tasks, and part of that work included running tests. If tests took longer than 5 minutes, the cache expired, and the whole thing was then being processed as uncached input.
I created a skill to better manage agents, including a way to keep the caches valid, and it has decreased my usage considerably. I now struggle to use my 100% each week and I hit claude hard. I have a reset expiring in 5 days and I'm trying to burn down the tokens (68% with 3d 8h left) quickly so I can use my reset before it expires.
Token efficiency just means understanding the way the system works. Best person to help with that... my old friend claude.
After I stopped letting Claude run tests, my token usage went from "gone by Wednesday" to "what can I automate this Saturday so my tokens won't go to waste".
Thanks this is very helpful, i didn't realize that. But not sure what to do since running tests is how Claude often catches mistakes and iterates. If i run them myself, the cache will still expire.
However where i think you have a point is that it tends to run tests quite casually where a reviewer would have been enough to catch mistakes.
IMHO, the real problem is the 5min cache way too small. They should up it to 1 hour.
What is the reason for this? Surely running tests by itself isn't necessarily token-expensive? And it provides cheap feedback for the agent: to see whether they're breaking things.
My guess is that the LLM (usually Sonnet or Opus in my case) is burning tokens polling to see if the output finished - and there is often a lot of it. My current project is Rust, Python, and Kotlin for what it's worth.
I don't actually know the real mechanism, I came across this serendipitously.
.NET solution having 120+ projects, with 7000 tests of which maybe 1500 are integration and/or component tests.
Buid alone takes 4 minutes. Test suite heavily parallelized runs in about 2 to 3 minutes, infrastructure dependencies fully containerized using podman, no external API call.
And this is without mentioning the 5000 frontend tests (angular), including Playwright end-to-end tests wired to the backend (about 3 to 4 minutes run without the e2e tests).
And this is running on a dev machine with 64gb of RAM and an NVMe disk.
Some systems are just massive, and even if you run focused tests, it's easy for a change to have a large blast at least requiring a large rebuild, and at times to rerun a larger set of tests.
I just told it that I want to run the tests, because so much of the code was vibed at that point and I felt I was losing understanding of the application. A few days of that and I could see that my token use was much lower than usual. Going back to LLM-run testing returned the high token usage.
Yeah, I should see if I can create a similar skill. The tests do spend some time to complete.
Claude: Confirmed: 195M of the 212M cache tokens (92%) come from full re-writes after 5+ minute gaps, accounting for 77% of relative cost. Now I'll check the existing wrappers so the skill
builds on them.
And I immediately turned it off because that just invites utter churn and stupidity. Part of that is just CC though, their compaction is miles behind Codex.
Not OP, but I typically ask it to report status of whatever is running; it can run a background task that reports to it every N minutes. With Claude Code in VS Code and a Pro subscription, it seems like the one-hour cache is the default, making it an issue less often.
I have some fairly large personal software projects and I still don't understand what useful work people are getting out of burning all these tokens on unattended agents.
What exactly are they doing for you that's giving you $200 of value in such a short timeframe?
As a dev who hasn't used LLMs as an embedded part of their work, and uncertain about it all, i'd be curious to know:
- How long do you reckon it would take a human to do the task?
- What percentage of features/ feature-extensions you add would have been done at all if you didn't have agentic assistance? Is there a tension between llm productivity gains and 'because we can' feature additions?
- Are many of the tasks for throwaway v1 features, exploratory tries?
- How speced out is the work you give it, eg did this task have pages of specs, how long did the spec take to write?
- How deeply will you manually review the work and how much time will it take? Will you read all the code / review at a high level and interrogate with llm / have the llm review and self iterate?
For what I am currently doing, years, if not a decade or maybe even two. Claude has already been running for 3 months constantly working on this project. I've been trying to do this several times before, but the scope outgrew my available time/life, too big for one person. But now I can actually build a production release by just setting the direction and let Claude to all the wiring, plumbing, tests etc.
I do now add all the major features I want, and write a task list where I add a few more minor tasks each week that just come to my mind. I need to write them down before I forget. I often ask Claude to implement several different implementations/algorithms and compare them. There are plenty of trade-offs, which usually end up in building configuration profiles, so the application can work optimally in multiple environments.
The specification is actually really large, and has grown over time as I've been learning and new ideas has come to my mind. I do not review/audit the code, it's massive would take me months, no chance! I do however use code inspection/audit tools like TSAN, ASAN and more. I will also spend time on full code reviews with the coming Fable 5.5 when a release is about ready. I do however review behavior together with Claude, and this is also where the specification grow, because this and that should be possible, but not this and that. User stories also helps.
Keep doing what you're doing, this guy clearly is just revving his engine because he likes the way it sounds and can afford the gasoline. I wouldnt be asking people like this for advise, they're completely off the rails.
There's no reason to blow through tokens this way, there's a 99% chance nobody uses the output (except maybe themselves).
I swear I have the same question in my head always when I read such statements! I have never once reached 90% that includes everyday usage across multiple repos, artefact generation, device debugging, long running harness work!
I am not saying OP is not telling the truth. I just really want to understand!
It’s because they don’t know what they’re doing. It’s like when openAI was using 10,000 agent clusters to solve problems… then realised the model could do the same work on its own and solve the same problem.
Anyone who tells you they have a “team of subagents” and “code reviewer” agents and “project manager” agents doesn’t know what they are doing. That’s how they burn thousands of dollars in a day.
Do you have a source for OpenAI discovering they can solve Navier Stokes with a single agent instead of a 10k swarm?
I think I’ll trust the experience of the people in the labs who have unlimited token budgets and are running experiments continuously over some anonymous rando slinging mud on the internet.
Also my own experience, which says having code reviewer agents is one billion percent useful. That comment alone completely disqualifies you as a reliable source of info.
I decided to try something a little weird research wise the other day analyzing about a thousand older research papers and burned through my $100 chatgpt sub + 3 resets in three hours and only finished half of them. It's the first time I've done that, though.
Not OP, but I used to run out in a few days, for me it was a matter of suboptimal code review practices with many subagents looking at different aspects, often expanded to look at more than the diff.
I can drain a $200 OAI plan in 2 hours flat working on my 10 projects simultaneously.
Maybe you're not being ambitious enough with these things? They are at their most fun when you spec out sprawling architecture documents and watch 1+8 astras buzzsaw through it.
I'm only at 260B for the year, though. I'm sure someone who has used more will be along shortly to chime in
E: The other subthread talking about tests is correct. Expensive models should not waste a single second waiting on tests or validation. If you cannot assign a single cheap model to be the test bitch, then just tell them to frontload everything and clean up later as a single wave.
Not meant as a bad faith response but have you shipped anything? Do you have income from this? Why 10 projects apart from fueling ADHD and "never finishing a project"?
Not OP and while I’m not draining my accounts in a single day/hour, I’m hitting limits with a $200-Claude and $100-OpenAI account.
> Have you shipping anything?
Yes, hundreds of PRs and shipped new features to 3 different platforms (day job and 2 side-business apps/platforms I’ve been developing for many years, pre-LLM). Those new features have gone out to customers and been used.
> Do you have income from this?
Well my day job for one, and I’ve been able to more easily sell to new clients with my side business. Without the changes I’ve shipped I would have been able to land a couple of the clients I’ve signed on recently.
I don’t think OP’s message implied they are never finishing a project, maybe they edited it?
In addition to the 3 income-generating projects, I have a number of other “just for me” projects. Some are mature now and only need occasional development when I want a new feature. Some are in flux as I build them out (like my “software factory”).
At this point I’m limited by compute and tokens more than anything else.
Does work pay for your subscription? I think many workplaces would worry about personal accounts for work projects.
EDIT: but thanks for a useful answer to OP's question. As a programming dabbler I also find it hard to imagine how to push much past the 5hr limit on $20USD subscription.
Its easy when you spawn a couple of subagents in a homelab with a rack with 5+ servers.
This is how one of my claude code session looks like right now, one prompt from yesterday:
Not sure if this will help you but it helped me. I was using subagents and workflows for lots of long running tasks, and part of that work included running tests. If tests took longer than 5 minutes, the cache expired, and the whole thing was then being processed as uncached input.
I created a skill to better manage agents, including a way to keep the caches valid, and it has decreased my usage considerably. I now struggle to use my 100% each week and I hit claude hard. I have a reset expiring in 5 days and I'm trying to burn down the tokens (68% with 3d 8h left) quickly so I can use my reset before it expires.
Token efficiency just means understanding the way the system works. Best person to help with that... my old friend claude.
After I stopped letting Claude run tests, my token usage went from "gone by Wednesday" to "what can I automate this Saturday so my tokens won't go to waste".
Thanks this is very helpful, i didn't realize that. But not sure what to do since running tests is how Claude often catches mistakes and iterates. If i run them myself, the cache will still expire.
However where i think you have a point is that it tends to run tests quite casually where a reviewer would have been enough to catch mistakes.
IMHO, the real problem is the 5min cache way too small. They should up it to 1 hour.
What is the reason for this? Surely running tests by itself isn't necessarily token-expensive? And it provides cheap feedback for the agent: to see whether they're breaking things.
My guess is that the LLM (usually Sonnet or Opus in my case) is burning tokens polling to see if the output finished - and there is often a lot of it. My current project is Rust, Python, and Kotlin for what it's worth.
I don't actually know the real mechanism, I came across this serendipitously.
By default Anthropic's token cache TTL is 5 minutes. If your tests take >5m to execute, when the LLM resumes your entire context is a cache miss
There are many reasons why your test suite shouldn't take >5m to execute. This is just one of them.
.NET solution having 120+ projects, with 7000 tests of which maybe 1500 are integration and/or component tests.
Buid alone takes 4 minutes. Test suite heavily parallelized runs in about 2 to 3 minutes, infrastructure dependencies fully containerized using podman, no external API call.
And this is without mentioning the 5000 frontend tests (angular), including Playwright end-to-end tests wired to the backend (about 3 to 4 minutes run without the e2e tests).
And this is running on a dev machine with 64gb of RAM and an NVMe disk.
Some systems are just massive, and even if you run focused tests, it's easy for a change to have a large blast at least requiring a large rebuild, and at times to rerun a larger set of tests.
I've worked on projects where the test suite would take hours.
Do you have a link to the docs for that? Thanks.
How do you do that?
I just told it that I want to run the tests, because so much of the code was vibed at that point and I felt I was losing understanding of the application. A few days of that and I could see that my token use was much lower than usual. Going back to LLM-run testing returned the high token usage.
Yeah, I should see if I can create a similar skill. The tests do spend some time to complete.
Claude: Confirmed: 195M of the 212M cache tokens (92%) come from full re-writes after 5+ minute gaps, accounting for 77% of relative cost. Now I'll check the existing wrappers so the skill builds on them.
Yesterday in my CC there was an update that added a feature to auto compact immediately prior to the cache being invalidated.
And I immediately turned it off because that just invites utter churn and stupidity. Part of that is just CC though, their compaction is miles behind Codex.
Some to think about with both Claude and skills.
Can you invert things. What’s deterministic and what needs “thinking”?
Can you create a script that runs things chaining together the deterministic parts with calling out to Claude for glue that needs “thinking”.
Don't suppose you would be happy to share the skill? Or is it fairly unique to your work flow?
I think the tests problem is really having an impact for me. Hadn't considered it before so thanks for flagging.
How do you keep the caches valid?
Not OP, but I typically ask it to report status of whatever is running; it can run a background task that reports to it every N minutes. With Claude Code in VS Code and a Pro subscription, it seems like the one-hour cache is the default, making it an issue less often.
Ask Claude, it's pretty wild to see.
> including a way to keep the caches valid
Maybe doing bogus small requests to prevent the cache from expiring?
I mean in the API you can set the cache expiration time right? But I guess claude code can't do this
> my old friend claude.
Awfully new old friend.
Google is the ex, Altavista is the high school sweetheart who tragically passed at a young age.
I have some fairly large personal software projects and I still don't understand what useful work people are getting out of burning all these tokens on unattended agents.
What exactly are they doing for you that's giving you $200 of value in such a short timeframe?
As a dev who hasn't used LLMs as an embedded part of their work, and uncertain about it all, i'd be curious to know:
- How long do you reckon it would take a human to do the task?
- What percentage of features/ feature-extensions you add would have been done at all if you didn't have agentic assistance? Is there a tension between llm productivity gains and 'because we can' feature additions?
- Are many of the tasks for throwaway v1 features, exploratory tries?
- How speced out is the work you give it, eg did this task have pages of specs, how long did the spec take to write?
- How deeply will you manually review the work and how much time will it take? Will you read all the code / review at a high level and interrogate with llm / have the llm review and self iterate?
For what I am currently doing, years, if not a decade or maybe even two. Claude has already been running for 3 months constantly working on this project. I've been trying to do this several times before, but the scope outgrew my available time/life, too big for one person. But now I can actually build a production release by just setting the direction and let Claude to all the wiring, plumbing, tests etc.
I do now add all the major features I want, and write a task list where I add a few more minor tasks each week that just come to my mind. I need to write them down before I forget. I often ask Claude to implement several different implementations/algorithms and compare them. There are plenty of trade-offs, which usually end up in building configuration profiles, so the application can work optimally in multiple environments.
The specification is actually really large, and has grown over time as I've been learning and new ideas has come to my mind. I do not review/audit the code, it's massive would take me months, no chance! I do however use code inspection/audit tools like TSAN, ASAN and more. I will also spend time on full code reviews with the coming Fable 5.5 when a release is about ready. I do however review behavior together with Claude, and this is also where the specification grow, because this and that should be possible, but not this and that. User stories also helps.
Thanks, appreciate the reply.
Keep doing what you're doing, this guy clearly is just revving his engine because he likes the way it sounds and can afford the gasoline. I wouldnt be asking people like this for advise, they're completely off the rails.
There's no reason to blow through tokens this way, there's a 99% chance nobody uses the output (except maybe themselves).
Yes, but what was the task that they were working on? This tells nothing.
That’s the “how” we’re wondering about.
[flagged]
You're doing this wrong.
I swear I have the same question in my head always when I read such statements! I have never once reached 90% that includes everyday usage across multiple repos, artefact generation, device debugging, long running harness work!
I am not saying OP is not telling the truth. I just really want to understand!
It’s because they don’t know what they’re doing. It’s like when openAI was using 10,000 agent clusters to solve problems… then realised the model could do the same work on its own and solve the same problem.
Anyone who tells you they have a “team of subagents” and “code reviewer” agents and “project manager” agents doesn’t know what they are doing. That’s how they burn thousands of dollars in a day.
Do you have a source for OpenAI discovering they can solve Navier Stokes with a single agent instead of a 10k swarm?
I think I’ll trust the experience of the people in the labs who have unlimited token budgets and are running experiments continuously over some anonymous rando slinging mud on the internet.
Also my own experience, which says having code reviewer agents is one billion percent useful. That comment alone completely disqualifies you as a reliable source of info.
I decided to try something a little weird research wise the other day analyzing about a thousand older research papers and burned through my $100 chatgpt sub + 3 resets in three hours and only finished half of them. It's the first time I've done that, though.
in my experience there's a huge difference in token usage when creating new projects versus working on existing ones
Not OP, but I used to run out in a few days, for me it was a matter of suboptimal code review practices with many subagents looking at different aspects, often expanded to look at more than the diff.
skill issues, brute forcing with AI.
not reviewing its ongoing work and spawning subagents that end up looping each other
Loading tons of MCPs and never compacting
[dead]
[dead]
I can drain a $200 OAI plan in 2 hours flat working on my 10 projects simultaneously.
Maybe you're not being ambitious enough with these things? They are at their most fun when you spec out sprawling architecture documents and watch 1+8 astras buzzsaw through it.
I'm only at 260B for the year, though. I'm sure someone who has used more will be along shortly to chime in
E: The other subthread talking about tests is correct. Expensive models should not waste a single second waiting on tests or validation. If you cannot assign a single cheap model to be the test bitch, then just tell them to frontload everything and clean up later as a single wave.
Not meant as a bad faith response but have you shipped anything? Do you have income from this? Why 10 projects apart from fueling ADHD and "never finishing a project"?
Not OP and while I’m not draining my accounts in a single day/hour, I’m hitting limits with a $200-Claude and $100-OpenAI account.
> Have you shipping anything?
Yes, hundreds of PRs and shipped new features to 3 different platforms (day job and 2 side-business apps/platforms I’ve been developing for many years, pre-LLM). Those new features have gone out to customers and been used.
> Do you have income from this?
Well my day job for one, and I’ve been able to more easily sell to new clients with my side business. Without the changes I’ve shipped I would have been able to land a couple of the clients I’ve signed on recently.
I don’t think OP’s message implied they are never finishing a project, maybe they edited it?
In addition to the 3 income-generating projects, I have a number of other “just for me” projects. Some are mature now and only need occasional development when I want a new feature. Some are in flux as I build them out (like my “software factory”).
At this point I’m limited by compute and tokens more than anything else.
Does work pay for your subscription? I think many workplaces would worry about personal accounts for work projects.
EDIT: but thanks for a useful answer to OP's question. As a programming dabbler I also find it hard to imagine how to push much past the 5hr limit on $20USD subscription.
Work provides a stipend/reimbursement of $100, the rest I pay for.
It’s paid for itself though, at least from my side business income.
Some of us want to build and don't care that it doesn't monetise.
Surely such people does not have 100usd/hour budget?
Would love to see these 10 projects that are being worked on simultaneously