Still need 2 or 3 days to properly quant this. But it is really good model to run if you have 64GiB or larger RAM on Mac. On M5 Ultra, you get ~3400 tok/s for prompt processing and ~50 tok/s for token generation, quite usable state.
Still need 2 or 3 days to properly quant this. But it is really good model to run if you have 64GiB or larger RAM on Mac. On M5 Ultra, you get ~3400 tok/s for prompt processing and ~50 tok/s for token generation, quite usable state.
Forgive the ignorance, but how are you running a 763B parameter model on 64GB of RAM?
SSD weights streaming. The actual weights is 552B, then engram.
which engine?
https://releases.drawthings.ai/p/public-beta-of-local-code-b...
By “quantitizing” it
The smallest quant is still over 100GB, so I don't know how that fits in 64GB RAM. It would have to be partly streamed from disk.