Can anyone answer me: What is the hardware cost to run a single instance of a frontier model? I've heard everything from $25K to $3M.
I think it comes down to: Will the hardware cost for frontier models come down?
Also, here's a business idea: Web crawler for open weight models. You just pay a monthly fee so that your home model can continuously update on a large web crawl each hour/day.
> What is the hardware cost to run a single instance of a frontier model?
The latest Kimi model, which scores higher than 5.1 Fable on Artificial Analysis, is estimated to require around 6-7 TB of VRAM to run. If we are going with many A100 GPUs for this, then we are looking around a million dollars in GPUs alone
What? We know exactly what it takes to run: 2.8 TB VRAM for FP8...1.4 TB VRAM for FP4.
A 3T frontier model can be run on 4 maxxed out Mac studios. But only for a single user.
I imagine that there's a lot of variance there based on what frontier model you want to run. Besides that, there's probably economies of scale that make it difficult to really ask how much it costs to run a single instance.
What do you consider a frontier model + what speeds do you want them to run at? These two questions need to be answered to answer your question.