My main question is whether when put into practical use, this can be measured in tokens/second, or more like 1 token per minute... I have seen locally hosted LLM that are as slow as 1 tok/second still be very useful if you give it a project to do something overnight and metaphorically walk away from it, check back with what it has done in 6 or 8 hours.
0.05 to 0.1 tok/s on the other hand, as reported in the URL for the lowest class of hardware, isn't really usable for much.
edit: I think this is a fantastic project in general concept, and look forward to seeing more efforts towards the general idea of being able to run a 350B to 900B size model locally, even if as slow as 1 tok/s, on hardware that ordinary people can afford. Anything along the general concept of "we have fast read NVME SSD storage, we have a big ass model on local disk, we'll read it at 11GB/tok as we need it, not try to load the whole thing".
The funny thing is Claude Cowork has taught me to be patient with response timelines. I’m now figuring I’ll be running locally no later than 2028.
(I want to spend no more than $10k. And I want to run a model comparable to today’s SOTA.)
For 10k you can buy a used dual socket Intel or amd based rackmount server with a terabyte of ram, and run models on cpu only at a reasonable speed. Same server would have been 4-5k a couple years ago before ram price rise.
Or buy one on eBay with 512GB that has half its slots populated and then buy the matching 512GB kit to add.
Which CPU gen are you suggesting, is there any writeup on such setup where <10K (not incl. power bill) cpu only rig is giving usable token speeds on latest SoTA open weights models?
In my experience with rig half that cost, entire exercise of running coding models locally has been a huge disappointment.
Cost/Value when compared to cloud services is just not there, but I see the merit for those who value privacy over quality of output and want a backup of huge condensed corpus of data within their control.
Kudos to OP though, They had clear goals and they achieved it.
I realized I didn't answer the CPU question, as a very quickly chosen example from eBay, there's a Dell R740XD with two Xeon Gold 6254 CPUs, 768GB RAM for sale for something like $5799 USD right now. I'm sure if I put some more time into it I could piece together something with a full terabyte for around the same price. Or faster/better CPUs, more core count CPUs by buying the system with no RAM, or minimal RAM (64GB) and then adding the DIMM kits from the more reputable refurb server part vendors on ebay.
It won't be fast at all, for certain, but it'll have enough memory to prove a configuration and be able to really use gargantuan GGUF format LLMs in the latest compiled llama-server. Re: electricity, I pay the equivalent of $0.07 ro $0.09 USD per kWh so it's not an extreme burden to have a theoretical 500W server running. Something like $35 to $50 of electricity a month if it's 500W 24x7.
Xeon Scalable in general seems like a good idea due to 6-channel (relatively) inexpensive RDIMM memory, but I've been reading that NUMA kills inference performance. Anyone got experience with multi-socket systems? IIRC even within the socket these cpus are divided into sub-numa nodes.
Even though LLM benchmarks are very opinionated, I would really like to see some numbers for the setup parent suggested. From what I read elsewhere, anything below $40K in HW costs is not worth the effort for coding models locally.
The old Cascade Lake based server found by the previous poster is still new enough to have instructions for relatively fast AI inference with the INT8 format.
So for optimal speed the models must be quantized in this format.
It is very likely that with INT8 models those CPUs are fast enough so that the inference throughput is limited by the memory bandwidth (384-bit interface to DDR4-2933 per socket, i.e. 282 GB/s for both sockets).
The memory throughput for such an old server is very similar to an AMD Ryzen Strix Halo, NVIDIA DGX Spark or Apple M5 Pro, but it has much more memory.
The inference speed should be very similar to those, but with bigger LLMs.
Would be nice if you could somehow connect GPU-levels of parallel floating point cores to that amount of memory. I guess that's what the big AI datacenters are doing, but how can we do that on a budget?
I think there is a good sized population of people who absolutely don't want to submit everything they do to an off site service, or let their content be used for unknown training purposes, and will tolerate slowness at 1 to 10 tok/s as a tradeoff.
Or people who want or need to run an uncensored (abliterated) gguf file to deal with controversial topics that a paid LLM service will refuse to work with or ban you for.
Not just controversial but also regulated areas. Virtually every law firm would be interested on locally-hosted AI at a reasonable price. So too ever medical research lab. Every CGI firm doing work for film/TV. And all the video game developers.
Do they care about locally-hosted, or only about self-hosted? I'm not really clear why a local box would be any better than running on a private AWS instance in any of these scenarios...
For one, doing the math on what it costs to rent a 768GB+ RAM AWS system with 40+ high performance CPU cores makes it very unappealing to pay for 12, 24, 36 months of it.
The largest high performance compute ec2 offering, the c9g.metal-48xl , maxes out at 384GB RAM and already costs a shitload.
The m9gd.48xlarge and m9gd.metal-48xl both have 768GB RAM and I cringe to think what they cost monthly. I just did the math on one of these and it costs $12 per hour, or $289 a day, or $8900+ for one month.
Also plenty of Europeans or people from other locations may consider it as an unacceptable risk factor to put their "off site" self hosted AI stuff with an American controlled company. Particularly if the servers are physically in the USA.
Hetzner will also rent you 768 GB of RAM with a Blackwell 6000 Max Q GPU for €2300/month [1].
Yes, it's a boatload of cash, but that's a €13,000 GPU and €20,000 of RAM at present prices. There is a segment of businesses where a fixed €28k/year bill is going to be preferred over plonking down €40k for a (theoretically) depreciating asset and ongoing colocation costs.
[1]: https://www.hetzner.com/dedicated-rootserver/gex131/
Renting something at a rate that'd be purchased in less than 2 years seems very myopic to me. And yeah it depreciates, but not to zero. So if you're speaking of the breakeven point after liquidation, you're probably there in well under a year at those rental prices.
> Renting something at a rate that'd be purchased in less than 2 years seems very myopic to me
And yet basically all AWS customers are doing exactly that. Turns out that making CAPEX "someone else's problem" is worth quite a lot to many businesses
that would be implying that "private" really means anything for AWS. Because if it's "private" as in "private" github repos that were totally not used for training copilot because they said so or "private" claude chats that are totally scanned even if you have enterprise contracts to check you are not doing anything malicious or are from china or whatever, and this will totally not be used for training...
can we trust any US based service to guarantee privacy and confidentiality? especially to us european frienemies?
> that would be implying that "private" really means anything for AWS
Insert your dedicated hosting provider of choice for 'AWS' (somewhere like Hetzner will be cheaper anyway).
But in general, AWS hosts are yours, running your code, with your security policies enforced. Sure, the US government can silently subpoena the contents thereof, but aside from that fairly extreme case, it's not like AWS is handing your data over to 3rd parties.
I would suspect that one would buy based on mem-bus & PCIe bus speeds more than CPU for this, and just dial down the CPU parameters to save power. Most of the time and power will be consumed by memory and bus transfers because the CPU will mostly be waiting to the right set of weights and factors to multiply.
[flagged]
> (I want to spend no more than $10k. And I want to run a model comparable to today’s SOTA.)
The question is, will you want to run a model comparable to today's (meaning 2026) SOTA in 2028? Humans always want the latest shiny LLM model.
Today's SOTA also sounds totally sufficient to me, but I wonder how much our standards will inflate by 2028. Maybe a lot, maybe not at all...very hard to say.
This seems to vary by person. I get immense value in coding assistance from Qwen 3.6 35B-A3B which is like a frontier model from a year ago. But a lot of people say it’s stupid, useless, a toy, etc. I do work by the “short leash” method and mainly just use the model for brainstorming/planning/design assistance and zipping through the drudgery of boilerplate and executing refactors. I don’t think this tier of model is good for “hey LLM, build me a Github clone” ... but I also don’t see the value in that use anyway.
Could you expand more on what you do with qwen3.6? Because I couldn't get the denser 27B version to do trivial "take this pattern, repeat it over a single file with minimal thought, just slightly beyond what I can do with sed" reliably.
Certainly. First of all, I am using OpenCode as the harness. (I have heard there are better harnesses such as little-coder for small open-weights models, but I haven't tried them yet.) Looking over some of my recent sessions, here are some examples:
- Asking Qwen to review project docs (requirements, user stories, etc) so that "we" can evaluate an iterate on an API design. Then back-and-forth chat about possible design directions. Then I ask for a rough-sketch plan of the one I'm interested in. I provide some tweaks to the plan and request a final plan in full detail. I switch to build mode and say go; everything is written to spec.
- Asking Qwen to write a suite of tests covering X, Y, Z issues with permutations A, B, C per issue.
- Asking Qwen to edit the shape of a CNN to insert auxiliary branches for intermediate supervision, and to extract out part of the network as a modular component with parameterized architecture.
I have less experience with the dense 27B because it's too slow to use on Apple Silicon. But regardless of which model you try, I would recommend trying a full-fat cloud hosted version of it first, so that you can get a sense of what it's capable of when the inference stack is correctly configured. LLMs are very sensitive to quantization formats, discrepancies in chat templates, etc. That kind of stuff is make-or-break.
How was qwen3.6 launched?
The thing is, everyone has their own variant of "qwen3.6 27b" depending on the launch parameters, ranging from "SOTA in its class" to "completely broken"
Caveat: I have not been able to try that model locally, so no personal experience. Running this locally at usable speeds would be cost prohibitive for personal coding use for me.
But if we can believe you that it's doing what a Claude model was doing a year ago then I'd say: OMG no I really never want to go back to that level of frustration getting an agent to do what I want it to do.
> OMG no I really never want to go back to that level of frustration getting an agent to do what I want it to do.
While it probably won't matter enough to change your mind, remember that you've gotten better at extracting value from all models than you were a year ago - plus the harnesses and other tools have gotten a lot better too.
> I don’t think this tier of model is good for “hey LLM, build me a Github clone” ... but I also don’t see the value in that use anyway.
What could be more valuable than outputting the exact thing you asked for?
Because the thing you get, from a prompt like that - even with a sota llm like fable - is a Potemkin village.
Knowing what to ask for, for one. Nobody can just whip up a specification for a system that satisfies all of the technical/design/business constraints that will turn out to have been relevant, has good usability for the target users, hits the right performance tradeoffs - all out of thin air. If anyone could, THAT would be priceless.
Looking at how critical we are about today’s models, vs where we were last year, and I don’t expect anyone to be content with Fable-class models in 2028.
Expectations seem to be rising at a faster rate than models can improve.
I’ve been wondering if chat is the wrong interface for slower local models (and some projects) and maybe something like a ticket system is a better fit. I just decided how I would test this idea on my available hardware before I go drop money on a Mac Studio or GPUs. I’ll probably have a POC this week. There is nothing novel here, just need to spend the time to get it working for me.
Having a thin python/ts orchestrator and workers that pick up tasks from the directories like events and decide whether to make deterministic calls and wait is pretty standard albeit custom way of doing things in this space where you're bottlenecked by the concurrent call your workers/agents can make.
The hard thing is always keeping complexity low and being ZeroOps.
Are there any frameworks/scaffolding/harnesses or general resources on this you can share? I’d love to learn more
Most any ticketing system can integrate with ordinary IMAP and smtp email flow, so you can really use any agent that can "do" inbound and outbound email to talk to a self hosted ticket queue.
I no longer use a harness directly, instead I use Github issues/Linear to work on multiple tickets in parallel while the agents are doing work:
https://github.com/skorokithakis/symphony
So I’ve been thinking about this problem a lot, specifically as it relates to running LLMs at home, and I’ve been using GLM-5.2 to make an SMTP/IMAP-to-LLM gateway.
https://tangled.org/clee.sh/posthorn
I’ve been wondering about something similar - a system that enforces (or does the heavy lifting) of dividing a large task into smaller sub-tasks so that it’s easy to run/check/test each one independently - even on a fresh model instance if needed.
This is based on the observation that the medium-sized open weight models (~20-35b) are very able to one-shot smaller discrete tasks but seem to lose their way project managing themselves through larger tasks that have multiple steps.
This is actually really smart. It would be like working with a team of humans.
I have a 3 Mac Studio set up and built an IDE / harness (propelcode.app) and would be interested in contributing if you’re open to collaboration
Time to make EmailGPT
Use the ticket system built into mininote.ink 's mcp server. Works perfect right out of the box. Also great notetaking app.
Docs:
https://mininote.ink/docs/mcp-docs
Now many mini-PCs and desktops are able to read simultaneously from 1 PCIe 5.0 SSD and 1 PCIe 4.0 SSD. This can ensure a reading throughput around 20 GB/s, i.e. 20 times faster than on author's system.
With only 1 PCIe 5.0 SSD, the reading throughput is still significantly more than 10 times faster than on author's system.
So it is likely that inference speeds around 1 token/s are achievable on something like a NUC mini-PC.
In the readme you can see benchmark which everyone with different hardware is running Colibrì, and I have to say I've seen great times! I'm always doing more to improve!
I have a 16-core system with 256GB RAM here I could try it with but regretfully it's so old the CPUs aren't AVX2 capable. Otherwise it makes a fairly good llama-server test system for CPU only stuff. Oh well. Time to upgrade (painful to the wallet these days).
Maybe we can see some integration!
If you get good at extracting remarkable performance from the most lesser of instruments enough to pull their own weight regardless, just imagine what it can be like when such a practitioner gets behind the keyboard of a world-class Steinway. And just does what they do best. Without ever having touched such a capable instrument themself.
On a level playing field the expression of virtuosity can outshine those who have never known any instrumental limitations at all :)
When pulling way more than your own weight happens like for few others.
There should be an award for getting the most out of the electronics rather than trying to reach orbit by building the tallest pile of e-waste.
First Prize right before your eyes !
Grande praise !
And just starting to ascend toward an unconquered summit that others find forbidding ;) Or they find uninteresting since the limit naturally lies on firm earth somewhere below the stratosphere.
Thanks for kind words!
Agreed!
0.05 to 0.1 per sec could still be quite useful if it was the speed for inferring a whole batch of tokens concurrently. Of course this actually requires fairly good SSD read performance (since you need to read a sizeable fraction of the complete model at every token batch in order to get good reuse) and is ultimately limited by CPU/GPU thermals which are a tight constraint on typical inference platforms. It's also only really feasible with tiny KV caches, which requires either a very small context or sticking to KV-cache efficient models such as the DeepSeek V4 series. Still, this might be one way of making use of existing lower-end hardware for practical inference of non-tiny models.
For most projects the more practical solution is to use clouds offering GLM 5.2 for free. 1 token per minute is minuscule compared to their rate limits for free usage.
But it's about the journey not the destination. My current running local LLMs train of thought...
> on hardware that ordinary people can afford
These days, can "ordinary people" afford 24GB of ram and half a TB of NVME ssd?
sigh
> These days, can "ordinary people" afford 24GB of ram and half a TB of NVME ssd?
You can, right now, buy a brand new Mini-PC at or above this spec for $600 at retail [1]
Of course, if you want it in a desktop format with a much faster CPU, its going to cost you more.
[1]: https://www.amazon.com/GMKtec-M6-Ultra-Upgraded-Computers/dp...
After 18y of thinkpads, this year I bouth a Lenovo yoga for... Cheap (1000€).
32G RAM, nvme 1TB, core ultra 258V.
Looking at the prices now... Wow, was I lucky.
Tried some of the 7b models locally, more than usable, around 30token/sec, not with the NPU, but using the ARC integrated GPU.
I am a noob for this, but I guess it's time to experiment more with this local setup
The very boring pair of two 16GB ddr5 6000 I had in my newegg shopping cart went from $399 to $475, so increasingly the answer will be "no".
I bought whole Intel N100 mini pc with 16GB of DDR5 in it in 2023 for $AUD289 (so about $US200). I got a 16GB (DDR4) SODIMM in 2022 for $AUD88 ($US60).
Does it have to be DDR5? Is the limit RAM speed, or SSD speed?
I was just using that as an example of constant on going price rises, it was the most mundane and not particularly fast ddr5 6000 stuff. The 6400 is even more ridiculous.
Maybe that's a measure of the self-fulfilling dollar incentive toward "renting" someone else's RAM in the future rather than trying to actually own such an outlandishly luxury item :\
Maybe not afford new, but they probably already had it from before the current crisis?
Ideally this engineer's approach will yield better performance on lesser equipment in the future, if they keep up the good work after they get more-capable gear to experiment with as time goes by :)
[flagged]