This roughly lines up with my personal experience that in March a combination of stronger models and better tooling on my end let me start running jobs unattended 24/7 (using Anthropic sub and my own hardware). Their $8000/day per researcher spend is crazy though, I'm curious how they keep track of the work.
Sounds like OpenAI are in the token-maxxing camp, so who knows what individual employees are doing to work their way up the leaderboard?
If you spend $8000 to generate an animated pelican riding a bike, then how much tracking does it really need?
Is the guy who spent $300,000 or so translating the FLT proof to Lean going to get a big Christmas bonus?
That was Anthropic.
End of day, output and results are top target of measurements, token consumption is the obvious number that they would like to disclose for their own business benefits and a simple metrics that correlate with the output.
Rest assured, capitalist appears irrational in wasting money, but they certainly care more about profit.
Taking a profit means you have to show numbers and the sooner you show numbers the harder it is to take people’s money.
Can you elaborate on this? Especially the tooling.
I tried something similar and I remember it was still pretty dodgy in February.
Pretty much goal + task + dependency infrastructure to help avoid drift during long runs, especially across compaction boundaries. I have spent a lot of time doing automation with models at various strengths including some of the early open-weights models (Llama, Mistral, etc) so I have a pretty good feel for how to steer productively, there isn't any deep magic just scaffolding built out of reading a lot of traces and debugging stuck agents.
my stack in a sentence: refine the docs/prompts/skills often, that's your biggest job, use both frontier labs models reviewing each other, don't solve individual problems only the systemic ones (set standards strategically, don't define tactics)
If I had that many tokens/dollars I would be running canaries and adversarial verification in prod based on e.g. traffic replay, live fuzzing, all kinds of things to build confidence without direct human line-by-line review. If I had $100k to spend next month I could probably get through it, I'm running $2500+-api-equivalent a week at this point and I feel very token limited. Will be time for a 2nd or 3rd subscription soon for both labs I think.
Fable was a revolution, still learning how best to use it, 5.1 felt like a notable upgrade. At this point I launch a workflow with 10-20 minutes of interactive setup (and even that I feel might be too much), it runs for hours, and the PR is trivially mergeable (I still review every line, but 95% are just merge, maybe 4% are feedback needed, 1% are thrown away and regenerated, which implies I'm being insufficiently ambitious)
Thanks!
> If I had that many tokens/dollars I would be running canaries and adversarial verification in prod based on e.g. traffic replay, live fuzzing, all kinds of things to build confidence without direct human line-by-line review.
This part jumped out at me. There's something to watch out for here.
I recently had a funny experience. I delegated a major feature to an agent.
It turned out that it had implemented it precisely backwards, in a way which was pointless and which made things worse.
But it had written countless tests for the feature and all the tests were green.
I realised in that moment that even formal verification would not have helped, because it would simply have written a mathematical proof of the correctness of the incorrect feature...
Yes, you need some kind of other source of truth. I think the best way to get that is to do clean room development with a different agent, but ultimately if you give them the wrong idea they'll do the wrong thing.
The other thing I do, not as much as I should, but it's very powerful, is to generate spikes and deliberately throw them away to understand how to prompt better. Like I generated a swift version of the react native app I'm working on, and Alloy provers for the state transitions. None of it is production quality but getting great results that way is useful to scope future work.
If you've managed people, these are all familiar problems. I found you need much more than a functional specification, you also need motivation, background, related work, ideas tried, etc., because those help disambiguate the right path in the inevitable situation where your original task description is unclear or conflicts with itself.
Yes I'm using the entire consultancy stack - define values, etc, and work your way down the "where do these not match reality on the ground and need change", but for little robot people instead of (arguably less messy) humans.
> I think the best way to get that is to do clean room development with a different agent
What is this about? Could you give an example?
That would be the mother of all circular accounting: the main clients of OpenAI are OpenAI employees.
These researchers are paid millions of dollars for their work. I doubt trust is really an issue at that level.
Yes, because no employee with million-dollar comp has ever been untrustworthy in the history of business.
Imagine if one of the humans at OpenAI was misaligned! We should get the AI to research this possibility once they've been aligned.
> let me start running jobs unattended 24/7 (using Anthropic sub and my own hardware)
How are you running jobs unattended 24/7 without hitting your token limits?
Similar to hgoel the model is managing the project's arc and writing + debugging code, but the underlying work is pretty compute intensive and all LLM output that is part of the final product is generated by local LLMs. Claude Code builds the pipeline that does the work, the pipeline runs fully on open-weights models and is reproducible top to bottom. There is a lot of detail to cost management. First is of course the Claude Code subscription is heavily discounted vs API costs. Then managing context size and turn count, which multiplied are basically what determine usage accounting (cached read is almost all of the cost). Auto compaction at 175k or 200k tokens (model the right number for your work), sub-agents with good model selection, tools to predict subtask difficulty so the sub-agents are correctly sized to complete under the compaction limit. Lots of focus on tooling to improve turn efficiency (e.g. the tilth utility by another user here for querying code). This started as a few scripts in one of my research projects but now is how I run all of my agent coding workspaces, and in another month will probably start replacing Claude Code itself for my purposes.
It depends on the time the job itself takes. If you're having the LLM handle a training run for another model, the LLM is probably spending most of its time waiting for iterations rather than consuming tokens.
For a task I left a local model running on overnight, only ~100k tokens were used because most of the time was just waiting on tests to finish, then waking up, tweaking a few settings and trying again.
I'm currently running two 24/7 semi-autonomous AI research projects using Fable 5.1. It's on track to burn through my weekly quota in about 3 days. I check progress in the morning and in the evening, and provide some light steering.
See my sibling comment, you can probably robo-code yourself some tooling to alleviate a lot of that in a few hours but if you want help shoot me an e-mail. I'm interested in seeing other people's workflows.
My only experience in >24h agents is with economically sane models (one of GLM5.2, 5.3-flash for orchestration, DSV4-flash for implementation, and glm5.3|sol|kimi3 agents + subagents reviewing at the end)
Over 24h my token spend is <30$. Excluding tokens for review it's <10$. With the absurdly gigantic subscription subsidies and a reasonable workflow I suspect one could run parallel agents.
I'm not sure what the point would be though unless working on some kind of optimization problem -- it takes me days to review <24h of the agent's output. It's almost always near enough to correct to be shippable; though I do give it feedback and iterate until it's better than the code I would have written.
It only really makes sense for problems that are complex and require iterations that don't themselves require much review. E.g. if you want find, PoC, and patch bugs, the output can be reviewed without reading all the traces. Or if you want to write a custom tool that does some job using local LLMs, assembling that pipeline, tuning the prompts, etc takes a long time but reading the final tests + eval data + code is enough to get a lot of confidence that it works right. Model checkers can help too, for example I wanted multi-sink Bluetooth audio support in Gnome for my kids so I hooked the hardware up and robo-coded the core logic specifically to be checkable with Kani.
>don't themselves require much review
I'm still wary of any unreviewed code - though my area of work is not tolerant of defects.
Agree on targets / verifiable indications of progress or success being a prerequisite for this being useful - although that covers quite a lot of SWE work.
This sounds like more work than just writing the code yourself. You'll say it isn't. I don't believe you.
That is an incorrect presumption - I think it's plausible this is more work; it's certainly far more taxing.
I'm at a point in my career where a small minority of my time is coding. The AIs can do in a day what would have taken me a week uninterrupted with acceptable (in some cases inferior prior to human feedback--but in some cases superior!) quality.
As I do not have 10 let alone 40 hours per week to devote to coding I think it increases the amount of high quality work product I can create with a given time investment. As I review it I merge small independent units and decompose the work.
All that is to say I don't really like it - but I suspect for most *well defined* coding tasks human produced code from highly experienced engineers will largely cease to exist in the next year -- getting cheap/relatively horrible models to produce good code is now straightforward.
OTOH I never use AI for any human facing communication outside of making my writing shorter. IMO AI slop "documents" are almost certainly a drag on organizational productivity.
/loop ?
I suspect the $8000/day figure is the equivalent in API costs. But I also suspect gross margin on their API rates are 80-90%
[flagged]