Use the 27b, it's better in every way once you add MTP (which speeds up dense models but often doesn't add any performance to MoE models like the 35b-a3b). I get around 100TK/s on my 2x 3090 machine and 85 on my M5 Max.
Use the 27b, it's better in every way once you add MTP (which speeds up dense models but often doesn't add any performance to MoE models like the 35b-a3b). I get around 100TK/s on my 2x 3090 machine and 85 on my M5 Max.
Thanks, I'll give it a go.
(I generally find standard 27B too slow to enjoy using, whereas 35B-A3B is pretty snappy.)