Yes, the tunnel vision around local models on this site is crazy. The percentage of people in the world who can afford the hardware is extremely low.
Yes, the tunnel vision around local models on this site is crazy. The percentage of people in the world who can afford the hardware is extremely low.
It seems roughly similar to the pricing level of personal computers in the early eighties (i.e. IBM PC and Apple Macintosh). I’d expect prices to come down significantly over the next few years. Not so much in the next year or two, but after that.
That's in nominal dollars. However, inflation since then has been about a factor of ten to fifty and it hasn't trickled down at all.
Just like for warships, the complexity and cost of building cutting edge hardware has grown exponentially up to a point where a significant chunk of the world's computing is dependent on 2 companies: ASML, TSMC. We shouldn't extrapolate linearly from examples from the 80s.
No, but I wouldn’t expect it to stagnate like with Intel in the 2010s either. Maybe the biggest caveat is that most people will be fine with using cloud providers, so the market for non-server hardware won’t be subject to as much competition.
There's a non-small contingent who lucked into the periodic games machine upgrade at the right time to snag a {3,4,5}090 rig just before everything exploded. It's a small contingent now but it was less so then. And now those people can add a second card for roughly what that whole system would have cost new originally.
The current supply chain problems will eventually pass.
So will the stock of 3090s, as well as their ability to run contemporary local models. And there will be no supply of newer equivalents of those GPUs, because NVIDIA has since wisened up, and is using the very capabilities you need for local LLMs as market segment differentiator.
I don't think there is tunnel vision. I'm just saying that I have a couple 3090s I invested in a handful of years ago, and they are still going strong today as multiple GPU-needing technologies emerged.
I'm not saying everyone has to run local LLMs, because the APIs are in a race to the bottom, and my $10 of OpenRouter credits I bought months ago is down to $8.94 because most models give you MILLIONS of tokens for a US Quarter.
> I'm just saying that I have a couple 3090s
This is tunnel vision. The percentage of people who could afford the hardware you could at the time you back it so vanishingly small. I do not know a single non-tech person who has multiple graphics cards in a single computer.
And right now the demand for GPU is far outpacing the supply, even with factories at full production, which is keeping prices high and out of reach of most people. But unless something happens to shut down the factories (not impossible, but hasn't happened yet), eventually production will catch up to demand and prices will return to sane-ish levels. Won't happen this year, almost certainly not next year... but I would be shocked if the current high prices were to persist for a decade. Eventually the percentage of people who can afford that hardware will grow to be a decent chunk of the computer-owning population. And they'll be following READMEs written by the early adopters, for installing open-source harnesses to work with open-weight models.
My personal expectation is closer to 5 years than 10, which is why I wouldn't touch Anthropic or OpenAI stock with a ten-foot pole, personally, no matter how high their theoretical valuation is. Because their business model is doomed in the long run.
Newer cards aimed at consumer market are not capable of being used for local models the way 3090s are. That's on purpose: this capability is now used to price-differentiate between "normies playing games" and "companies in data center business".
It's a decreasing pool as well I'd say, the dev machine I built last summer would make less financial sense to me now for example.
I mean, I'm not rushing out to buy that kind of hardware myself, but it is a matter of perspective. People commonly spend an order of magnitude more on a car, and that's just the sticker price.