Can't wait for the DwarfStar quants - I have been using DeepSeek v4 flash (preview) as my main coding agent for months now (running on my 128gb mbp) - it seems this model outperforms GLM 5.2 on nearly every metric. Thanks for sharing the news, I was refreshing huggingface but gave up thinking it likely would take some more time.
I actually run it as a server - so most of the time I don't have to listen to it right next to me - it's just sitting in another room in my house - but I often am traveling with it and will have it sitting right next to my coding laptop and yea the fan runs non-stop - it's not obnoxious so i can pretty easily tune it out - also airpods/noise canceling headphones help!
There’s fan noise, but the acoustic engineers at Apple have done a very good job of making it be pretty much not noticeable to someone who has noise sensitivity (me).
the noise is pretty doable, and so is the heat, I find. I read these messages before I had the machine and expected far worse. The tok/s is for me the dealbreaker as I prefer to have multiple sessions. For nightly runs i do like it a lot, or to be a node in a mesh.
Generally get 20-25tps - prefill is pretty good around 400-450tps. I have been using compaction at around 100k tokens but mostly just cause it was the default in pi coding agent - might see if i can expand it a bit.
the unsloth GGUF at <165GB will run on most 256GB RAM pure-CPU systems (or with llama-server and a mix of loading as much as you can onto a single 32GB, 48GB or 96GB GPU and the rest onto system DRAM).
Google TPUs are built around a 128×128 systolic array of multiply-accumulate (MAC) units. Trainium 1, Trainium 2, and Inferentia 2 also feature a 128x128 systolic array.
You learn something every day. Today, it was the term "systolic array": A systolic array is a specialized grid of simple, interconnected processing units designed to execute parallel data operations—like matrix multiplication—by rhythmically passing data directly from cell to neighboring cell without writing intermediate results back to main memory.
The term comes from the biological word systole (the contraction of the heart pumping blood through the body). In a systolic array, data "pulses" through a network of processing elements on every clock cycle, driven by a global clock beat.
Just because scales are grouped by 128x128 tiles, does not mean you need a single compute tile that large. It works completely fine to process it with multiple smaller tiles that get given the same scales, like how this works on Hopper and Blackwell today
Blackwell. You could do it with wgmma on hooper but then you'll only be able to run 1 CTA per SM. There are cases where this is OK, but most commonly 128x128 mmas are primitive in tcgen05.mma i.e blackwell. The fundamental reasons is that the systolic array accelerator (TMA) until blackwell wrote to the cuda core registers themselves so you were limited by the register file size of the SM[SP]. In blackwell there's a separate TMEM where the mma unit stores it's output.
You can even do higher 256x256 and such using a hardware feature called 2-CTA MMA, this is essentially them letting neighbouring pairs of SMs co-operate and access each other's memory.
As for sibling comments, huawei ascend is more of an NPU-style architecture where you can easily have much bigger MMAs as primitive. But you usually don't anyways for many reasons.
Can't wait for the DwarfStar quants - I have been using DeepSeek v4 flash (preview) as my main coding agent for months now (running on my 128gb mbp) - it seems this model outperforms GLM 5.2 on nearly every metric. Thanks for sharing the news, I was refreshing huggingface but gave up thinking it likely would take some more time.
are you working in earplugs? :)) even with 128 gigs of ram it must be super noisy.
I actually run it as a server - so most of the time I don't have to listen to it right next to me - it's just sitting in another room in my house - but I often am traveling with it and will have it sitting right next to my coding laptop and yea the fan runs non-stop - it's not obnoxious so i can pretty easily tune it out - also airpods/noise canceling headphones help!
There’s fan noise, but the acoustic engineers at Apple have done a very good job of making it be pretty much not noticeable to someone who has noise sensitivity (me).
the noise is pretty doable, and so is the heat, I find. I read these messages before I had the machine and expected far worse. The tok/s is for me the dealbreaker as I prefer to have multiple sessions. For nightly runs i do like it a lot, or to be a node in a mesh.
my AC is noiser than my GPU server.
I've put Opus to it and it says it will take 3-4h to do the process to the new weights. hoping it works!
you don’t need new weight. try vllm-moet from github. it will autogenerate 2-bit plane.
What kind of tps are you getting?
Generally get 20-25tps - prefill is pretty good around 400-450tps. I have been using compaction at around 100k tokens but mostly just cause it was the default in pi coding agent - might see if i can expand it a bit.
I've had it run to ~400k when debugging "obscure" (to it) sequences. Wouldn't recommend more.
same. been running it in a dgx spark and it slaps
the unsloth GGUF at <165GB will run on most 256GB RAM pure-CPU systems (or with llama-server and a mix of loading as much as you can onto a single 32GB, 48GB or 96GB GPU and the rest onto system DRAM).
https://huggingface.co/unsloth/DeepSeek-V4-Flash-0731-GGUF
Hmm, it's targeting a HW accelerator with a 128x128 matmul primitive, which one is that? Warp Group Matrix Multiply Accumulate on H100?
Google TPUs are built around a 128×128 systolic array of multiply-accumulate (MAC) units. Trainium 1, Trainium 2, and Inferentia 2 also feature a 128x128 systolic array.
You learn something every day. Today, it was the term "systolic array": A systolic array is a specialized grid of simple, interconnected processing units designed to execute parallel data operations—like matrix multiplication—by rhythmically passing data directly from cell to neighboring cell without writing intermediate results back to main memory.
The term comes from the biological word systole (the contraction of the heart pumping blood through the body). In a systolic array, data "pulses" through a network of processing elements on every clock cycle, driven by a global clock beat.
TIL. TY.
Just because scales are grouped by 128x128 tiles, does not mean you need a single compute tile that large. It works completely fine to process it with multiple smaller tiles that get given the same scales, like how this works on Hopper and Blackwell today
Huawei Ascend NPU
Blackwell. You could do it with wgmma on hooper but then you'll only be able to run 1 CTA per SM. There are cases where this is OK, but most commonly 128x128 mmas are primitive in tcgen05.mma i.e blackwell. The fundamental reasons is that the systolic array accelerator (TMA) until blackwell wrote to the cuda core registers themselves so you were limited by the register file size of the SM[SP]. In blackwell there's a separate TMEM where the mma unit stores it's output. You can even do higher 256x256 and such using a hardware feature called 2-CTA MMA, this is essentially them letting neighbouring pairs of SMs co-operate and access each other's memory.
As for sibling comments, huawei ascend is more of an NPU-style architecture where you can easily have much bigger MMAs as primitive. But you usually don't anyways for many reasons.
Huawei Ascend presumably?