We desperately need to keep local models competitive with the hyper scalers. I won't want to be beholden to them and their whims forever. With anthropic now restricting game reverse engineering I think we're about to cross a Rubicon of censorship.
We desperately need to keep local models competitive with the hyper scalers. I won't want to be beholden to them and their whims forever. With anthropic now restricting game reverse engineering I think we're about to cross a Rubicon of censorship.
The nice thing is that current local modes don't go away. The weights are out. Qwen 27b and deepseek are already very useful. Hardware will keep getting stronger so after the current memory crisis you can run them more cheaply.
Of course the proprietary models will continue to improve and we have no guarantee that there will be new open models. But for coding, the local models are already good enough for a vast amount of use cases, if you are an experienced developer yourself and can guide it.
You don't need bigger or better models. Current models are more than adequate for any task.
The key thing is tool use.
The model is fixed but tool use introduces new information into the system, so once you have a model that is good enough to effectively use the full suite of tools available (debug, test, profile, etc) and write code to effectively combine them and synthesize their results for a particular task, there is nothing that it can't solve.
New models are continually ingesting new information from usage, and embedding the results from people running this process back into the base model, which makes it more efficient, gets more things right the first time, etc, but there is no step gap in functional outcomes left.
Computing and programming are a limited domain with a small fixed set of constructs that can be applied in an infinite number of combinations. Nothing ever closes the gap between the ability of the model to automate the assembly of these constructs and the necessity of expert human direction to pick what system to build out of the infinite range of possible systems.
Frontier models can keep producing very expensive one-shot demos to pitch to investors but they are never going to replace engineers.
I think the future should be efficient quantization which can make model with the same level as Qwen 3.8 27b now accessible from our laptop.
The future is already here :)
These 3bit ISTA quants of qwen are supposed to be 95% as accurate as the full model and runs moothly on laptops:
https://huggingface.co/ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF
There are multiple independent tests of this on YouTube.
They aren't "Just as accurate". What you see with these quants is the providers only ever show performance on already saturated benchmarks. You'll see labs put out 1-bit quants Deepseek v4.1 claiming only 1% degradation because they're only showing the benchmarks which tell that story. Then you look at the performance on those same benchmarks for significantly weaker models like Qwen3.8-27B even and it's virtually the same because that benchmark is saturated.
I like these quants as much as the next guy when they're all I can run but I have both R9700 with 32GB VRAM and a 5060 Ti with only 16GB VRAM and so I run both Q6 Unlsoth quants and ISTA IQ3_XXS quants and yeah there is an absolute world of difference in realworld performance.
IMHO I think it's pretty scummy behavior from the quant providers to frame these quants this way. Just be honest about where the degradation is really occurring on the non-saturated benchmarks.
This is a really good point that goes the other way as well. Some use cases are great for smaller and cheaper models, but it's not obviously when that's the case so I can bank the savings.
All I know is that Strata and these models are the most I have ever gotten out of my machine. I get 70 t/s on a standard gaming rig 9070 XT / 5900X / 64 GB DDR4 @3400.
I don't know, it hallucinates quite badly - confuses js with python.
Apparently the ud-q4_k_xl version of qwen 3.8 27b is quite better than the q3 and not much larger. At least per r/localllama. (I'm also trying to find a good local model).
This is mostly a memory bandwidth issue. Qwen 3.8 27b and the related Bonsai 2 27B run on my M5 Max laptop with 36GB of ram. It’s a little slow due to my specific model having lower memory bandwidth than higher end models with more ram and cores. MoE models perform better here and expert streaming engines can allow larger than memory models to run.
Multiple teams and efforts are now making Qwen 3.8 27b work much faster on Mac and CUDA and the inference speed is doubling almost every week. There's now ongoing competition about who will make it work faster with exactly the same byte output at https://mlx.fast . TensorFold is the project to follow.
So it's not about the model (eg TensorFold/Qwen3.8-27B-MLX-4bit) it's about the inference engine?
Its possible (though very hard to do well, and needs a fair bit of compute) to improve the local models we have, too. Continued training is possible as time goes on.
>The nice thing is that current local modes don't go away.
Given pace of advance I'm not convinced this is a winning strategy. The models of today (open or not) are way smarter than those of a year ago.
...so we really do need open models to keep pace, not just be statically competitive at a given point in time
Even if Deepseek v4.1 is the last open source LLM I have access to, I would be perfectly fine with that. My secret is that I have this relatively unknown technique for solving problems where I use this quantum engine that developed from just atoms over 4.54 billion years into the most sophisticated machine in the known universe. It's a frontier model that every human has access too, actually.
So when I really want to solve a problem, I just think on it. And when I need something to iterate through the things I thought of quickly, I use deepseek.
> Even if Deepseek v4.1 is the last open source LLM I have access to, I would be perfectly fine with that.
Your quantum engine aside, current open-source LLMs will not age well. They have a knowledge cutoff. If you ask them five years from now about something that didn't exist when they were trained, chances are they won't be able to provide a useful answer. They may still be useful for many applications, but their value will eventually deteriorate. Having continuous access to up-to-date open-weight LLMs is therefore about much more than simply having access to "better" models.
we will soon figure out how to update the engrams with new information
Yes, but first we need to figure out how to collect that new information. An efficient and ruthless data harvesting in one of the LLM labs secrets.
Yep, fully satisfied with DS 4.1 Flash or GLM 5.3 Flash. I prefer using one chat at a time, actually thinking through things with it, as opposed to the YOLO run all night multi-agent thing. I mean I’m still okay with subagents if my main thread needs to explore a few options and then report back, but all on the same idea.
You might enjoy Cecli, the successor to Aider. It's made for the kind of workflow you are describing (full hands off yolo and recursive sub-agents also an option, but not the default)
cecli.dev
How are you running these? Not locally, I assume?
The thing here for me is not that I’m unable to solve problems with my brain, it’s that I value the automation in labor provided by coding agents. I do not want future advancements to be locked behind proprietary doors.
I agree with that but the cat is out of the bag now. Unless you believe in exponential improvement (which I don’t), the open models might be at a resource disadvantage, but that doesn’t mean that they can’t catch up or surpass the closed ones. Throwing money at a problem doesn’t guarantee success.
I mainly just want to point out the value of human ingenuity coupled with a model like DS4.1 is technically enough to get most people to where they want to go, and that includes smarter open models.
Then you probably don't want to be funding the companies that want and intend for it to be locked behind proprietary doors.
You say that now, but you don't know what the future models will be capable of. If models get ridiculously good, you might not be perfectly fine with that.
I think at that point I’ll be doing carpentry instead of computing.
How do you run DS 4.1 Flash if your computer isnt powerful and everything?
It can run on one strix halo machine, which is what I have. For everything else, there’s a vibrant community brute forcing more efficient engines and quantizations. There’s a minimum bar right now, so it depends on your hardware.
you have 5k$ of ram in a strix halo?
I bought my box for $3k last year because I hadn't updated my PC since the first threadripper 1950x, and the Strix Halo is the only x86 with unified memory that's optimized for the kind of CPU intensive stuff I do, and they finally have proper AVX512 too. It was the perfect upgrade for me (except for lack of PCIe 5 & ECC), which coincidentally happened to be perfectly positioned for local inferrence.
and I liked it so much (and saw that the RAM drought was going to last for at least a few years) so I bought a second one to use as a general server for myself and my family.
The fact that you point to the latest model as „I’d be fine with that“ proves my point.
If expectations were static you’d be fine with qwen 2.5…but nobody sane says that today.
It’s a moving target thus we need open weights to move
I’m all for local models and getting away from Anthropic/OAI but Qwen 27B might as well be a horse drawn carriage compared to a 2026 Porsche. Like, I guess it’s useful if you had nothing better but acting like it’s comparable to anything the frontier models have is just disingenuous.
disagree, it’s absolutely comparable. but an m5 max is where it shines. and there are 50 flags to finetune with llama server before you get it right. i do prefer llama server over mlx currently on mac as well.
It’s so poor for most tasks that you’re faster doing it manually.
DeepSeek and GLM are useful, though. Even the flash variants.
I have all three and you're not going to convince me of that.
Side-by-side Qwen 3.8 27B is ~6x as fast as DS4 and GLM 5.3 and faster than that if you get the speculative engine set up properly. That means you can iterate very rapidly on the same problem and for many problems this is the way to solve them. DS4 and GLM 5.3 shine when Qwen would take too many turns to solve something, they simply have more world knowledge and seem to home in on possible solutions faster in #tg but slower in wall clock time. If a particular problem is within striking range of all three then Qwen will usually be the fastest solver by far.
It also depends on how good your prompt is, whether you are setting up points for guidance or whether you expect everything to be solved in a single shot.
> Side-by-side Qwen 3.8 27B is ~6x as fast as DS4 and GLM 5.3
Wow, that sounds like a lot! Do you use/mean the regular models or flash variants here?
Specifically:
The GLM Quant is ~4 bpw, the DS4 is a Q8 and ninfer one is an NVFP4/BF16 I think. I run the ninfer one with ninfer-4090 on a hacked card with 48GB of VRAM. Max context for that one is 256K, DS4 has a 1M context and for GLM 5.3 it is 300K. The big models use all 8 GPUs.Qwen benefits in two ways: all of the VRAM is on a single GPU so there is no communications other than generated tokens and new activations, that gives pretty good utilization for the card, the second is of course simply the fact that it is a much smaller model so there is less to compute. The 4090 is still power limited so it could do more but it's loud (and hot) enough as it is.
More details here:
https://jacquesmattheij.com/local-inference-with-large-langu...
bro, you are looking at a repo that shows you exactly how to use it to achieve better results than a frontier model
But it's all for the good of humanity, so how can you complain?
https://forum.effectivealtruism.org/posts/53Gc35vDLK2u5nBxP/...
>Anthropic restricting game reverse engineering
Which is literally the coolest thing to come out of this...
Did they get a letter from Nintendo's legal team or something?
Maybe it was EA from my shenanigans :) https://dirtforever.net
It doesn’t matter, right? The open models should be capable of reversing anything prior to 2005, no?
> I won't want to be beholden to them and their whims forever.
Who's keeping you beholden to them now?
Wait and see, I give it fairly high odds everything tagged as 'uncensored' or 'abliterated' will get wiped from huggingface at some point in the future. The main barrier to wider spread adoption of this right now is that very few people have the legit on-prem hardware resources/budget to run something like an abliterated Q8 of deepseek v4.1 flash 1017 or mimo 2.6 or equivalent in size locally.
Doubt it. Even if HF drops them, it will not be hard for someone to move it to something like a decentralized torrent site dedicated to open source models.
Banning them would just invite competition which is why I think HF is holding out as long as they legally can.
Recently, HuggingFace has already taken down multiple models abliterated for cyber and with such descriptions, at their own initiative.
Example: https://huggingface.co/audnai/penclaw-GLM-5.3-abliterated-fo...
Previously, I recall HuggingFace only took models down if someone complained / sent a DMCA or cease and desist.
check https://pirateface.co/ for llms removed by hf
So they'll start branding them, "darknet models" or some such, and push for laws banning their use.
Punish the consumer, ala the copyright war since the 90s
some further info: https://www.reddit.com/r/LocalLLaMA/comments/1xd7ptj/models_...
>everything tagged as 'uncensored' or 'abliterated' will get wiped from huggingface at some point in the future.
Unlikely given how trivial it is to DIY this (I think...haven't myself but from afar it seems within avg hn crowd reach)
Think about it this way: if new open source models come restricted where they refuse to run tasks related to abliteration, then even if public utility programs exist, it will already add a lot of friction to how many people will be capable of doing it. Now, if unverified models become outlawed and will move to more obscure hosting sites, that will add friction as well.
Especially as people grow more dependent on agents doing all of their computer work.
Yes, it'll likely just result in some people standing up a more dedicated purpose community hosting site of abliterated/uncensored open weight GGUF files. In a theoretical scenario where HF bans abliterated/uncensored models I could very well see the same people who are producing those now simply moving the location they upload stuff to.
To use an old analogy, it's not rocket science:
https://github.com/p-e-w/heretic
I was able to remove much of the CCP censorship from Qwen 3.8 27B for around $90 using a LoRA. I multiplied the LoRA in permanently and listed it on HuggingFace.
The cost was approximately $80 for generating synthetic data and $10 to create and integrate the LoRA.
Shouldn't loras generally be distributed without being baked in the weights? Or is "de-baking" a lora trivial?
I distribute both https://huggingface.co/noisebridge/qwen3.8-27b-taiwan-pilled.
> I give it fairly high odds everything tagged as 'uncensored' or 'abliterated' will get wiped from huggingface at some point in the future
This is the kind of bet where I would take the opposite position without hesitation.
The only thing I could see them removing would be models that are specifically trained to do bad things, like a model trained to produce illicit underage image content. And I would be totally understanding of and encouraging of that, even if it makes the free speech absolutists unhappy.
I think the uncensored and abliterated models are more hype than actual threat. The journalists and politicians are too busy going after the AI labs every time someone does something bad and the police can find ChatGPT logs before it. I doubt we’re going to find a situation where someone was running local models and also keeping logs of everything and also commits a crime and also the journalists find it and also they make a story that catches on and so on.
It’s such a niche thing it’s just not a priority. There are some open LLM people who are treating it like a crisis before it happens, but come on. It’s the internet. We can download gigabytes of pirated TV shows despite that industry and lawmakers working hard to stop it for decades. If open models got banned somehow, you’d be able to download them the same way.
I didn't say they won't immediately move somewhere else or still be readily available. Just be taken off HF. It'll be a futile game of whack a mole for anyone trying ot stop it. The new corporate overlords at some point will face pressure. The 72GB 2160p HEVC copy of Apocalypse Now sitting on my home theatre file server which came from a well seeded torrent does indeed say that large file sizes of things that $bigcorp don't like will continue to be easily available.
Applying Heretic to your model weights is a very straightforward thing, though.
Hugging Face just hosts the models. Any hypothetical censorship on their end wouldn't stop you from running the original model provided directly by the builder.
> everything tagged as 'uncensored' or 'abliterated' will get wiped from huggingface
... you do know you can download from huggingface and make your own copy, right? And even if you couldn't, you could run heretic (or whatever approach they used) yourself on the official image. It's all just a well-understood process. A decent abliteration run on DS 4.1 flash will probably cost you about $25k.
They're not small but they're hardly out of reach. I have backups on my home NAS.
> very few people have the legit on-prem hardware resources/budget to run something like an abliterated Q8 of deepseek v4.1 flash 1017
It's something like $300-400/day for the raw hardware on RunPod, a little more on AWS. Not for home users, sure, but for a business wanting to eg check its security against the best of what attackers might bring, that's nothing. And it will go down.
The main barrier to wider spread adoption of this right now is that very few people have the legit on-prem hardware resources/budget to run something like an abliterated Q8 of deepseek v4.1 flash 1017
Not as bad as you might think. This costs about as much as a decent new car nowadays.
The abliterated 1017 release I've seen is crazy good.
I guess it depends if we're talking about decent new car at $40,000 or $95,000? I've seen lots of people claiming they're running v4.1 flash 1017 on local hardware but then it turns out it's something like the unsloth Q2 XXS or similar size. The Q8 is 508GB so assuming for a moment we're not running it on a Mac, we're looking at a cost of five or six 96GB GPUs to load the whole thing with a good sized full context and KV cache.
Haven't bothered with DS4.1 personally but I've seen reports of running it with native weights on 4x DGX Sparks. Currently ~$30k but last month that setup was ~$20k.
4x 96 GB runs the NVFP4 quants, which are indistinguishable from Q8 in real-world performance for this particular model. That's not always true but DeepSeek is unusually quantization-friendly. Also remember that a big chunk of the model consists of engrams that can live in system RAM or even on an SSD if necessary.
I imagine the various GGUFs are still pretty impressive, but haven't tried them myself.
> new car
...which is still tooo much for most people to spend on "AI hardware"....
More like, the ability to chain a demon in your basement that does what you tell it and can't be rug-pulled by his bosses in San Francisco.
AI is the endgame for the joke that computing is the act of etching arcane signs into crystal and imbuing them with lightning.
Recently saw a hand drawn cartoon to the effect of, a person is asking Claude or chatgpt to do something, it refuses, so the next step is "make the Chinese one do it", and the Chinese model is represented as a chained demon
Just gotta make sure that pentagram is rock-solid!
[flagged]
Yes it is extremely aggravating that Claude will often raise copyright objections even when doing something like putting images in a report for personal use. The AI companies are the most blatant copyright abusers in history.
> We
who is "we" exactly? because the vast majority of us (pretty much everybody except people working at at hyperscalers that are not the usual hyperscalers) can essentially do nothing.
I got banned from the Hermes discord today not gonna lie.
https://gnosh.ai/head-to-head.html
Did we ever get back to 16GB feasibility?
AFAIK the timeline was:
Any chance a good model will ever be seen in a consumer GPU again?Sonnet 5.5 refused to implement code using selenium/firefox to scrape a webpage on my laptop after we got a 403 from a cloud-based IP. I made that clanker generate datasets for the rest of the week to use up my weekly usage that would have been wasted.