There’s also input tokens. For many agentic use cases input/output ratio can be 2:1 or even 4:1. Non-quantised DeepSeek V4 Flash costs $0.14 / $0.28 on most inference providers with ZDR.
When self-hosting the model, cache hits are basically free. RAM cache also helps with hit rate (prefixes may be around for hours instead of minutes).
To be clear this project doesn’t aim to achieve the best inference economics per token. MI300X doesn’t have native MXFP4 so it’s not even the right platform for the model. That’s why very few deployment recipes are available.
It’s interesting to me because MI300X is quite accessible to a small team with budget for just 1-2 GPUs. DeepSeek V4 Flash otherwise wouldn’t even fit on 2x H100s.
We can run several coding agents during the day and batch inference jobs overnight and serve the entire team with guaranteed privacy, without compromising precision or speed.
In fact we found that many inference providers are quantising the weights or even KV cache, and due to the low prices they serve at massive batches, resulting in unstable throughput. I ran GSM8K as a quick validation test and this deployment is “better” than the OpenRouter endpoint in a statistically significant way (I wouldn’t name the provider here). I will run some follow up benchmarks and update the repo when I find some time.
830t/s is burst aggregate. ~500 is sustained and it's for 8 concurrent users. Meaning for $1.99/hour if you serve 8 users it's 8*$0.54, not just $0.54.
You shouldn't rent one out if you're just serving it for yourself, but from a financial standpoint if you sell to users you can take a 100% margin.
Bro 500 Aggregate. so that's 500 * 60 * 60 = 1.8M output which is .5$ at best... Not including pre-fill and stuff.
This is not the real margins, even if you are selling to 8 users it's 90 tps per median stream. So assuming that .6-.7$
This is not even remotely worth it.
You need to 3x this tps(~1500 tps) to be worth it, and that's what most providers are doing, at 20-30 users at 50-60 tps with better optimized batch processing and kernels you can make some profit.
Agentic workloads are somewhere around 1%/0.5%/98.5% input/output/cached tokens. Cached tokens are pretty much free for inference providers (if they implement sparse and compressed attention properly) and throughput for input tokens is much higher.
Lets assume that you've got 2 million input tokens, 1 million output tokens and 98.5 million cached tokens to process. That would cost 2 * $0.14 + 1 * $0.28 + 98.5 * $0.0028 = $0.8358 with DeepSeek API pricing.
For comparison, it would take 2M / 8000 + 1M / 800 = 1500 seconds to process this amount of tokens with the linked framework, which is about $0.83 when we assume $2/hr for one MI300X.
However, other inference providers have 10 times higher prices for cached tokens, which results in a comfortable margin.
And we should not discount that DeepSeek also gets paid in data, which is probably more valuable to them.
And I believe that this framework still has some room for optimization for generation with high batch sizes.
Your math is a bit funny if you're assuming the 1/0.5/98.5 ratios: you doubled input and output tokens but not cached. If you double cached tokens to match your original ratio it works out to around $1.11, and if you 10x the cached token cost it's around $6.08.
Based on your $0.83 estimate, the margin isn't great. This is within shooting distance of "at cost" which is probably pretty close to what DeepSeek is operating with, ignoring the value of the data they're collecting of course.
> And I believe that this framework still has some room for optimization for generation with high batch sizes.
If that optimization can bring this scenario closer to $0.50 then it gets pretty compelling, otherwise I'm not confident.
Oh, I messed up. Half-way through, I thought it would be a good idea to double the numbers so I don't have to deal with half millions, but forgot to also double the 98.5. Unfortunately, I can not edit it anymore.
I think the margins of DeepSeek may be a bit better than with this vibe-coded framework here, since they had the liberty of optimizing their models for their own hardware.
At the time, open frameworks were not anywhere close to achieving that number. Not sure whether they caught up. The software wizards at DeepSeek are quite skilled.
> should not discount that DeepSeek also gets paid in data, which is probably more valuable to them
That's agentic feedback loops for training, right? Any more detail on this, such as how they actually tell whether that data is good or not? That seems like a very hard problem, and like the value of that data is low compared to just building their own, controlled RL gyms.
Agents usually start with ingesting the existing code base, and DeepSeek can use those code bases for pretraining. And they will have filters on top of that to throw out garbage.
I am not sure how they are using the data for post-training, but there probably are ways to get signal out of it, e.g. sentiment analysis when the user begins cursing at the agent, or checking whether the user continued another session with the generated code, or started a new session with the same starting point as before, i.e. they git-stashed.
Generally, you can train on data that is quite bad (e.g. the entire internet). It will still work, but take much longer compared to clean data.
Definitely not. Inference is not as expensive to operate as many people seem to assume. The frontier labs are probably making a lot of money from selling tokens. It’s covering all of the R&D costs like salaries, collecting training material, and running the large training operations that costs a lot of money.
They claimed that OpenAI and Anthropic have positive gross margins. I don't think there are many credible claims saying that's not the case (at least for API usage)?
Use nvidia hardware instead and use a larger cluster serving many more users concurrently. Easily 10x–20x higher token rate per GPU with public solutions like dynamo and sglang.
I think Deepseek is selling roughly at cost (perhaps a slight premium). They don’t guarantee that they don’t train on the submitted prompts, so I suspect they are mining the data. Mining for what? Well, who knows. Best case, mining to make Deepseek better. That said, I use Deepseek all the time. It has done a whole lot of ‘ls’ commands on my system, though.
This is exactly what I came to say. The price of Flash is so cheap that trying to run it locally or with your own hardware is pointless. I was using it about a month ago to program some stuff and ran it for 4 days non-stop and it cost me about $2.
> trying to run it locally or with your own hardware is pointless.
Serving local models has advantages other than price. If you work in restricted industries, or have a strong need to protect your IP, or if you just value privacy more than cost, you now have options.
If you don't do any attention steering, custom decoding or meddle with the weights maybe. Services are worthless unless all you do is write positive prompts.
As others have mentioned, there's the privacy factor as well.
There’s also input tokens. For many agentic use cases input/output ratio can be 2:1 or even 4:1. Non-quantised DeepSeek V4 Flash costs $0.14 / $0.28 on most inference providers with ZDR. When self-hosting the model, cache hits are basically free. RAM cache also helps with hit rate (prefixes may be around for hours instead of minutes).
To be clear this project doesn’t aim to achieve the best inference economics per token. MI300X doesn’t have native MXFP4 so it’s not even the right platform for the model. That’s why very few deployment recipes are available.
It’s interesting to me because MI300X is quite accessible to a small team with budget for just 1-2 GPUs. DeepSeek V4 Flash otherwise wouldn’t even fit on 2x H100s.
We can run several coding agents during the day and batch inference jobs overnight and serve the entire team with guaranteed privacy, without compromising precision or speed.
In fact we found that many inference providers are quantising the weights or even KV cache, and due to the low prices they serve at massive batches, resulting in unstable throughput. I ran GSM8K as a quick validation test and this deployment is “better” than the OpenRouter endpoint in a statistically significant way (I wouldn’t name the provider here). I will run some follow up benchmarks and update the repo when I find some time.
830t/s is burst aggregate. ~500 is sustained and it's for 8 concurrent users. Meaning for $1.99/hour if you serve 8 users it's 8*$0.54, not just $0.54.
You shouldn't rent one out if you're just serving it for yourself, but from a financial standpoint if you sell to users you can take a 100% margin.
Bro 500 Aggregate. so that's 500 * 60 * 60 = 1.8M output which is .5$ at best... Not including pre-fill and stuff.
This is not the real margins, even if you are selling to 8 users it's 90 tps per median stream. So assuming that .6-.7$
This is not even remotely worth it.
You need to 3x this tps(~1500 tps) to be worth it, and that's what most providers are doing, at 20-30 users at 50-60 tps with better optimized batch processing and kernels you can make some profit.
You are right, it's not 500x8 it's 90x8.
You get privacy for 4 times the cost
How is that economically viable? They are selling at a loss?
Agentic workloads are somewhere around 1%/0.5%/98.5% input/output/cached tokens. Cached tokens are pretty much free for inference providers (if they implement sparse and compressed attention properly) and throughput for input tokens is much higher.
Lets assume that you've got 2 million input tokens, 1 million output tokens and 98.5 million cached tokens to process. That would cost 2 * $0.14 + 1 * $0.28 + 98.5 * $0.0028 = $0.8358 with DeepSeek API pricing.
For comparison, it would take 2M / 8000 + 1M / 800 = 1500 seconds to process this amount of tokens with the linked framework, which is about $0.83 when we assume $2/hr for one MI300X.
However, other inference providers have 10 times higher prices for cached tokens, which results in a comfortable margin.
And we should not discount that DeepSeek also gets paid in data, which is probably more valuable to them.
And I believe that this framework still has some room for optimization for generation with high batch sizes.
Your math is a bit funny if you're assuming the 1/0.5/98.5 ratios: you doubled input and output tokens but not cached. If you double cached tokens to match your original ratio it works out to around $1.11, and if you 10x the cached token cost it's around $6.08.
Based on your $0.83 estimate, the margin isn't great. This is within shooting distance of "at cost" which is probably pretty close to what DeepSeek is operating with, ignoring the value of the data they're collecting of course.
> And I believe that this framework still has some room for optimization for generation with high batch sizes.
If that optimization can bring this scenario closer to $0.50 then it gets pretty compelling, otherwise I'm not confident.
Oh, I messed up. Half-way through, I thought it would be a good idea to double the numbers so I don't have to deal with half millions, but forgot to also double the 98.5. Unfortunately, I can not edit it anymore.
I think the margins of DeepSeek may be a bit better than with this vibe-coded framework here, since they had the liberty of optimizing their models for their own hardware.
For DeepSeek V3, they claimed a cost profit margin of 545%: https://github.com/deepseek-ai/open-infra-index/blob/main/20...
At the time, open frameworks were not anywhere close to achieving that number. Not sure whether they caught up. The software wizards at DeepSeek are quite skilled.
> should not discount that DeepSeek also gets paid in data, which is probably more valuable to them
That's agentic feedback loops for training, right? Any more detail on this, such as how they actually tell whether that data is good or not? That seems like a very hard problem, and like the value of that data is low compared to just building their own, controlled RL gyms.
Agents usually start with ingesting the existing code base, and DeepSeek can use those code bases for pretraining. And they will have filters on top of that to throw out garbage.
I am not sure how they are using the data for post-training, but there probably are ways to get signal out of it, e.g. sentiment analysis when the user begins cursing at the agent, or checking whether the user continued another session with the generated code, or started a new session with the same starting point as before, i.e. they git-stashed.
Generally, you can train on data that is quite bad (e.g. the entire internet). It will still work, but take much longer compared to clean data.
> They are selling at a loss?
Definitely not. Inference is not as expensive to operate as many people seem to assume. The frontier labs are probably making a lot of money from selling tokens. It’s covering all of the R&D costs like salaries, collecting training material, and running the large training operations that costs a lot of money.
> It’s covering all of the R&D costs like salaries, collecting training material, and running the large training operations that costs a lot of money.
Are you claiming that the frontier labs like OpenAI and Anthropic are actually making a profit contrary to all the claims?
They claimed that OpenAI and Anthropic have positive gross margins. I don't think there are many credible claims saying that's not the case (at least for API usage)?
They claim their advantage is knowing how to serve their models efficiently, which is quite possible since they design for it.
Use nvidia hardware instead and use a larger cluster serving many more users concurrently. Easily 10x–20x higher token rate per GPU with public solutions like dynamo and sglang.
They get all our invaluable data which they'll use to train the next model, to get more data, to train the model after.
[flagged]
I think Deepseek is selling roughly at cost (perhaps a slight premium). They don’t guarantee that they don’t train on the submitted prompts, so I suspect they are mining the data. Mining for what? Well, who knows. Best case, mining to make Deepseek better. That said, I use Deepseek all the time. It has done a whole lot of ‘ls’ commands on my system, though.
This is exactly what I came to say. The price of Flash is so cheap that trying to run it locally or with your own hardware is pointless. I was using it about a month ago to program some stuff and ran it for 4 days non-stop and it cost me about $2.
It's expensive to run it locally at full quality, but at least on my setup, its about 5 times faster than any API, and is completely private.
> trying to run it locally or with your own hardware is pointless.
Serving local models has advantages other than price. If you work in restricted industries, or have a strong need to protect your IP, or if you just value privacy more than cost, you now have options.
[flagged]
If you don't do any attention steering, custom decoding or meddle with the weights maybe. Services are worthless unless all you do is write positive prompts.
As others have mentioned, there's the privacy factor as well.
With the cost of electricity, hardware depreciation and tok/s it rarely makes sense to run locally.