> if someone would kindly quantise this down for the 64GB paupers
The Q4_K_M is 75GB. I'm exactly at 64GB, and I wouldn't quantize it further. Instead, do partial weight residency and stream the rest from SSD.
> if someone would kindly quantise this down for the 64GB paupers
The Q4_K_M is 75GB. I'm exactly at 64GB, and I wouldn't quantize it further. Instead, do partial weight residency and stream the rest from SSD.