Open source models could be run by low-cost cloud providers, too. They could offer discounts for a long term contract and run it on dedicated hardware.
Open source models could be run by low-cost cloud providers, too. They could offer discounts for a long term contract and run it on dedicated hardware.
This. Your local LLM, even if shared between a pool of devs, is probably only going to be working 8 hours a day. Better to use a cloud provider, especially if you can find a way to ensure data security, if that is an issue for you.
Exactly. There is no shortage of providers hosting open source models with per-token pricing, with a variety of speeds and context sizes at different price points. Competition is strong and barriers of entry low, ensuring that margins stay low and prices fair.
If you want complete control over your data and don't trust anyone's assurances that they keep it private (and why should you) then you have to self-host. But if all you care about is a good price then the free market already provides that for open models
Hetzer already offers dedicated GPU servers from 180eur/month
Hetzner and Scaleway already do instances with GPUs so this kinda already exists
In fact, does anybody want to hire a server with me? I suspect it'll work out cheaper than Claude max etc: a server from hetzner starts at £220ish: https://www.hetzner.com/dedicated-rootserver/matrix-gpu/
It might be fun to work out how to share, too. A whole new breed of shell hosting.
I have a couple of non-GPU servers with them and quite a few Heztner Cloud projects, but I never understood their GPU offering. They have just two (small) VRAM sizes and you pay per month, whereas the ones like Runpod have a large selection of whatever you need, and they are cheaper, and you can rent them for a shorter period like two weeks, no setup time. Am I missing something?
There are use-case for smaller models that fit in 20gb VRAM, like in database sentiment analysis and such.
Sure, my point you can get these cheaper outside of Hetzner, so I don't really understand who are these for.