If I had to pick a product, I'd say an affordable 32GB mac would be the sweet spot for running local models that function well like Qwen 3.8.
It's true, most people don't run models, but being the default platform for running open weights seems like it has plenty of advantages right now. Just like sales benefited from developers defaulting to MacOS for most open source languages like Ruby, Go, Rust, and TypeScript.
32GB is not enough, it's unified/shared memory, you need to have space for usual system and user apps/services.
64GB+ or dedicated 48GB (2x24 on GPUs) is IMHO absolute minimum.
32GB of fast unified memory is enough for Qwen 3.8 27B.
- 16GB for the weights at Q4
- 9GB for the full 256K context at Q8
- 7GB spare for overhead and system.
The problem is that these Macs have 32GB of slow unified memory.
Edit: I'm thinking of a headless Mac mini, if you meant running it on the same machine you're using of course you'll need more memory, but LLMs are best served from a headless server so that's what I'd recommend.
Is this for setup for agentic coding? Why not also run the IDE compiler etc... on the same machine to use those CPU cores as well?
Keep in mind that if you want MTP it adds a few gigs. If you use sub-agents it turns already slow generation into even slower generation. Won't be doing any compling (so rust, c and probably go are not avaiable) becase those add memory pressure during compiling.
32gb of unified memory is enough enough for system to be used for anything other than LLM generation.
You can, you just need a beefier PC, and it's more annoying in terms of noise and heat vs throwing something on your server closet. Plus you don't need to worry about other software stealing resources and whatnot.
> Edit: I'm thinking of a headless Mac mini, if you meant running it on the same machine you're using of course you'll need more memory, but LLMs are best served from a headless server so that's what I'd recommend.
What? LLMs are best served from a massive PD disaggregated cluster of B300s connected via NVLink.
If you're running LLMs on a Mac Mini, it's because you want to run local, not because it's the best setup.
> If I had to pick a product, I'd say an affordable 32GB mac would be the sweet spot for running local models that function well like Qwen 3.8.
https://www.canirun.ai (five months ago: https://news.ycombinator.com/item?id=47363754 377 comments)
> If I had to pick a product, I'd say an affordable 32GB mac would be the sweet spot for running local models that function well like Qwen 3.8.
32GB is not enough RAM. I don't even own a device with less than 36GB at this point, and that device I only have because my employer is being cheap. 64GB is a reasonable starting point for running local LLMs + normal tasks. 128GB let's you really run most smaller models like Qwen 27B and 35BA3B with good context. Even Qwen3.8-Flash-Next runs in 128GB with a 4-bit quant.
32GB would be limited to running models like Gemma4 12B and smaller dense Qwen versions like 9B unless you were using very small quants which damages quality of response.
You are mistaken. I'm running Qwen 3.7 28B 4bit (MLX) with a 200k context window and everything total is 32GB RSS.
Is this the best? No. That's why I said the sweet spot. Getting from 16GB macs to 32GB is perhaps possible. Jumping to 64GB or 128GB as the default is simply unreasonable right now.
I have a similar machine, and briefly poked at running a local LLM, but got discouraged after a couple days. The quality, responsiveness, and impact on the rest of the system didn’t seem worth it to me.
What sorts of things are you doing with the local LLM? Anything interactive? Should I take another look?
Yes, 15-30 t/sec is pretty slow for local models so I recommend running local LLM tasks overnight where (vs paid plans) there isn't a risk of chewing through your token budget from a rogue loop or sub-agent. Even if it takes hours, you're sleeping anyway so no concern. herdr + pi works great for this but there are lots of harnesses.
Memory used : 38GB, and I haven't even started a LLM nor podman, I always fight with memory when using LLM on my mac with 48gb.
And I don't remember to have been able to have pushed to 200k context Qwen 3.6. 3.8 is running on my RTX 5090.
Qwen 27B runs very comfortably on a 5090. You need to use Q4 quants and Q8 KV cache. Here's the math
https://news.ycombinator.com/item?id=49514141
I assume you mean Qwen 3.8-27B? Yes, you can run this in 32GB of RAM, but it's very context limited. With KV cache compression and other techniques, it's better now than in the past, but I'd still want more RAM, personally.
EDIT to add that you need to reserve 8GB for the system if you don't want to cause problems on macOS, which means 32GB RAM = 24GB max for model + context. It takes 18-19GB to load a 4-bit quant of Qwen3.8-27B, so I'd be really surprised if you can actually get a 200k context window. You need to fit within a 24GB WSS (which is generally a more constrained RSS) to get stable performance on 32GB RAM.
I run Qwen 3.8 27B just fine on my Mac mini M4 24GB. I use Unsloth's Q3 XXS with 128k context. It successfully completes long horizon tasks with OpenCode.