The funny thing is Claude Cowork has taught me to be patient with response timelines. I’m now figuring I’ll be running locally no later than 2028.
(I want to spend no more than $10k. And I want to run a model comparable to today’s SOTA.)
The funny thing is Claude Cowork has taught me to be patient with response timelines. I’m now figuring I’ll be running locally no later than 2028.
(I want to spend no more than $10k. And I want to run a model comparable to today’s SOTA.)
For 10k you can buy a used dual socket Intel or amd based rackmount server with a terabyte of ram, and run models on cpu only at a reasonable speed. Same server would have been 4-5k a couple years ago before ram price rise.
Or buy one on eBay with 512GB that has half its slots populated and then buy the matching 512GB kit to add.
Which CPU gen are you suggesting, is there any writeup on such setup where <10K (not incl. power bill) cpu only rig is giving usable token speeds on latest SoTA open weights models?
In my experience with rig half that cost, entire exercise of running coding models locally has been a huge disappointment.
Cost/Value when compared to cloud services is just not there, but I see the merit for those who value privacy over quality of output and want a backup of huge condensed corpus of data within their control.
Kudos to OP though, They had clear goals and they achieved it.
I realized I didn't answer the CPU question, as a very quickly chosen example from eBay, there's a Dell R740XD with two Xeon Gold 6254 CPUs, 768GB RAM for sale for something like $5799 USD right now. I'm sure if I put some more time into it I could piece together something with a full terabyte for around the same price. Or faster/better CPUs, more core count CPUs by buying the system with no RAM, or minimal RAM (64GB) and then adding the DIMM kits from the more reputable refurb server part vendors on ebay.
It won't be fast at all, for certain, but it'll have enough memory to prove a configuration and be able to really use gargantuan GGUF format LLMs in the latest compiled llama-server. Re: electricity, I pay the equivalent of $0.07 ro $0.09 USD per kWh so it's not an extreme burden to have a theoretical 500W server running. Something like $35 to $50 of electricity a month if it's 500W 24x7.
Xeon Scalable in general seems like a good idea due to 6-channel (relatively) inexpensive RDIMM memory, but I've been reading that NUMA kills inference performance. Anyone got experience with multi-socket systems? IIRC even within the socket these cpus are divided into sub-numa nodes.
Even though LLM benchmarks are very opinionated, I would really like to see some numbers for the setup parent suggested. From what I read elsewhere, anything below $40K in HW costs is not worth the effort for coding models locally.
The old Cascade Lake based server found by the previous poster is still new enough to have instructions for relatively fast AI inference with the INT8 format.
So for optimal speed the models must be quantized in this format.
It is very likely that with INT8 models those CPUs are fast enough so that the inference throughput is limited by the memory bandwidth (384-bit interface to DDR4-2933 per socket, i.e. 282 GB/s for both sockets).
The memory throughput for such an old server is very similar to an AMD Ryzen Strix Halo, NVIDIA DGX Spark or Apple M5 Pro, but it has much more memory.
The inference speed should be very similar to those, but with bigger LLMs.
Would be nice if you could somehow connect GPU-levels of parallel floating point cores to that amount of memory. I guess that's what the big AI datacenters are doing, but how can we do that on a budget?
I think there is a good sized population of people who absolutely don't want to submit everything they do to an off site service, or let their content be used for unknown training purposes, and will tolerate slowness at 1 to 10 tok/s as a tradeoff.
Or people who want or need to run an uncensored (abliterated) gguf file to deal with controversial topics that a paid LLM service will refuse to work with or ban you for.
Not just controversial but also regulated areas. Virtually every law firm would be interested on locally-hosted AI at a reasonable price. So too ever medical research lab. Every CGI firm doing work for film/TV. And all the video game developers.
Do they care about locally-hosted, or only about self-hosted? I'm not really clear why a local box would be any better than running on a private AWS instance in any of these scenarios...
For one, doing the math on what it costs to rent a 768GB+ RAM AWS system with 40+ high performance CPU cores makes it very unappealing to pay for 12, 24, 36 months of it.
The largest high performance compute ec2 offering, the c9g.metal-48xl , maxes out at 384GB RAM and already costs a shitload.
The m9gd.48xlarge and m9gd.metal-48xl both have 768GB RAM and I cringe to think what they cost monthly. I just did the math on one of these and it costs $12 per hour, or $289 a day, or $8900+ for one month.
Also plenty of Europeans or people from other locations may consider it as an unacceptable risk factor to put their "off site" self hosted AI stuff with an American controlled company. Particularly if the servers are physically in the USA.
Hetzner will also rent you 768 GB of RAM with a Blackwell 6000 Max Q GPU for €2300/month [1].
Yes, it's a boatload of cash, but that's a €13,000 GPU and €20,000 of RAM at present prices. There is a segment of businesses where a fixed €28k/year bill is going to be preferred over plonking down €40k for a (theoretically) depreciating asset and ongoing colocation costs.
[1]: https://www.hetzner.com/dedicated-rootserver/gex131/
Renting something at a rate that'd be purchased in less than 2 years seems very myopic to me. And yeah it depreciates, but not to zero. So if you're speaking of the breakeven point after liquidation, you're probably there in well under a year at those rental prices.
> Renting something at a rate that'd be purchased in less than 2 years seems very myopic to me
And yet basically all AWS customers are doing exactly that. Turns out that making CAPEX "someone else's problem" is worth quite a lot to many businesses
that would be implying that "private" really means anything for AWS. Because if it's "private" as in "private" github repos that were totally not used for training copilot because they said so or "private" claude chats that are totally scanned even if you have enterprise contracts to check you are not doing anything malicious or are from china or whatever, and this will totally not be used for training...
can we trust any US based service to guarantee privacy and confidentiality? especially to us european frienemies?
> that would be implying that "private" really means anything for AWS
Insert your dedicated hosting provider of choice for 'AWS' (somewhere like Hetzner will be cheaper anyway).
But in general, AWS hosts are yours, running your code, with your security policies enforced. Sure, the US government can silently subpoena the contents thereof, but aside from that fairly extreme case, it's not like AWS is handing your data over to 3rd parties.
I would suspect that one would buy based on mem-bus & PCIe bus speeds more than CPU for this, and just dial down the CPU parameters to save power. Most of the time and power will be consumed by memory and bus transfers because the CPU will mostly be waiting to the right set of weights and factors to multiply.
[flagged]
> (I want to spend no more than $10k. And I want to run a model comparable to today’s SOTA.)
The question is, will you want to run a model comparable to today's (meaning 2026) SOTA in 2028? Humans always want the latest shiny LLM model.
Today's SOTA also sounds totally sufficient to me, but I wonder how much our standards will inflate by 2028. Maybe a lot, maybe not at all...very hard to say.
This seems to vary by person. I get immense value in coding assistance from Qwen 3.6 35B-A3B which is like a frontier model from a year ago. But a lot of people say it’s stupid, useless, a toy, etc. I do work by the “short leash” method and mainly just use the model for brainstorming/planning/design assistance and zipping through the drudgery of boilerplate and executing refactors. I don’t think this tier of model is good for “hey LLM, build me a Github clone” ... but I also don’t see the value in that use anyway.
Could you expand more on what you do with qwen3.6? Because I couldn't get the denser 27B version to do trivial "take this pattern, repeat it over a single file with minimal thought, just slightly beyond what I can do with sed" reliably.
Certainly. First of all, I am using OpenCode as the harness. (I have heard there are better harnesses such as little-coder for small open-weights models, but I haven't tried them yet.) Looking over some of my recent sessions, here are some examples:
- Asking Qwen to review project docs (requirements, user stories, etc) so that "we" can evaluate an iterate on an API design. Then back-and-forth chat about possible design directions. Then I ask for a rough-sketch plan of the one I'm interested in. I provide some tweaks to the plan and request a final plan in full detail. I switch to build mode and say go; everything is written to spec.
- Asking Qwen to write a suite of tests covering X, Y, Z issues with permutations A, B, C per issue.
- Asking Qwen to edit the shape of a CNN to insert auxiliary branches for intermediate supervision, and to extract out part of the network as a modular component with parameterized architecture.
I have less experience with the dense 27B because it's too slow to use on Apple Silicon. But regardless of which model you try, I would recommend trying a full-fat cloud hosted version of it first, so that you can get a sense of what it's capable of when the inference stack is correctly configured. LLMs are very sensitive to quantization formats, discrepancies in chat templates, etc. That kind of stuff is make-or-break.
How was qwen3.6 launched?
The thing is, everyone has their own variant of "qwen3.6 27b" depending on the launch parameters, ranging from "SOTA in its class" to "completely broken"
Caveat: I have not been able to try that model locally, so no personal experience. Running this locally at usable speeds would be cost prohibitive for personal coding use for me.
But if we can believe you that it's doing what a Claude model was doing a year ago then I'd say: OMG no I really never want to go back to that level of frustration getting an agent to do what I want it to do.
> OMG no I really never want to go back to that level of frustration getting an agent to do what I want it to do.
While it probably won't matter enough to change your mind, remember that you've gotten better at extracting value from all models than you were a year ago - plus the harnesses and other tools have gotten a lot better too.
> I don’t think this tier of model is good for “hey LLM, build me a Github clone” ... but I also don’t see the value in that use anyway.
What could be more valuable than outputting the exact thing you asked for?
Because the thing you get, from a prompt like that - even with a sota llm like fable - is a Potemkin village.
Knowing what to ask for, for one. Nobody can just whip up a specification for a system that satisfies all of the technical/design/business constraints that will turn out to have been relevant, has good usability for the target users, hits the right performance tradeoffs - all out of thin air. If anyone could, THAT would be priceless.
Looking at how critical we are about today’s models, vs where we were last year, and I don’t expect anyone to be content with Fable-class models in 2028.
Expectations seem to be rising at a faster rate than models can improve.