SSD streaming on an M5 Max 128GB: https://x.com/antirez/status/2082136334160818528
Soon decent speed across two Mac Studios with 512GB of RAM.
SSD streaming on an M5 Max 128GB: https://x.com/antirez/status/2082136334160818528
Soon decent speed across two Mac Studios with 512GB of RAM.
wow cool! I like watching the new models come out and how they end up crammed in to run on local machines. I learned what mxfp4 is thanks to this latest Kimi release - although it sounds like it means that there's less room for compression in the model compared to others.
Cool stuff. Do you have a the hardware and a way to bridge the compute? Or just hopeful?
0.3tx per second is decent speed?
And it gets worse with every token.
That's not what parent said, but this is already quite decent speed for unattended inference (overnight or even spanning multiple business days) which is arguably the right target for this model. This is a challenging model to infer locally, it has roughly ~115 GB of dense active parameters(!) plus ~25 GB of sparsely routed experts per token. Plus the KV cache (which is actually reasonably lean for this one model, around 27GB for a full 1Mtok context). What you're seeing in antirez's video is essentially the performance we should expect from a 128GiB system that has to load sparsely routed experts in full from disk because there's no real room for caching them.
192GiB Gorgon Halo systems will be an interesting future target for this model, the best you can do with 128GiB or less is probably to push batching higher in order to amortize the weights traffic over multiple inferences - which of course will sink single-session speeds even lower for a modest gain in total throughput.
Would it even finish over a few weeks?
For AI agents this would take a year
That's not what he said. He said with 2x the hardware it will be faster.
I’ve wondered for a while: given the lower cost of SSD per GB could you build a very wide RAID0 style striped array of SSDs (maybe one per slot) to get almost RAM like read speeds?
To really go fast you’d probably have to do PCB layout and do like 256 or 1024 chips in parallel with a fast SRAM aggregation buffer feeding a GPU or TPU rig.
Or could you do the same with custom layout of cheap slower RAM?
I wonder if anyone is doing this? You would flash in a model and then just run it. It would need RAM for context but much less of it.