Running LLMs locally for development doesn’t make sense to me. The hardware gets outdated in just a few years. Even hyperscalers replace their GPUs faster than they can buy them, plus the cost of running it locally, isn’t cheap. the cost saving just ain't there.
From the perspective of LLM inference, you currently mostly care about:
- Memory bandwidth; BUT the requirements are currently capped because models have stopped growing at around 1-1.5 trillion parameters for quite a while now. You only need more bandwidth if you're optimizing for the highest possible concurrency (i.e. you're a cloud provider). Also, MoE exists.
- Support for native low-precision math (like FP4 and FP8); BUT once your GPU supports native FP4 (Blackwell+), there's generally no reason for GPUs to go lower because of the obvious quality degradation.
- VRAM capacity - just like memory bandwidth, it's practically capped by 1-1.5 trillion parameter models and is unlikely to need much more in the near future. Also, the current trend is toward miniaturization: modern 30B-class models (which require far less VRAM), now completely destroy 200B-class models from just two years ago on most tasks. We also have better understanding now how to compress contexts.
Most model improvements currently seem to come from RL/harness-based methods, not from scaling models or running new algorithms that require fundamentally new GPUs.
So I don't see why GPUs that exist today must become "outdated" in a few years. They'll be seen as outdated by hyperscalers because they need to serve the maximum number of users as cheaply as possible, so of course they'll replace their GPUs with newer ones that have higher memory bandwidth or more tensor cores. But you don't need that for local inference.
3090 was released six years ago and is still very relevant for running models locally.
Qwen 3.6 35B runs on 32GB with a 1080. That GPU is from 2017.
> replace their GPUs faster than they can buy them
How does that work? They have negative GPUs now!