Is the 1/10+ reduction in memory usage applicable to larger models, i.e. would be possible to run a 200Gb model, by using 20Gb of memory?
Is the 1/10+ reduction in memory usage applicable to larger models, i.e. would be possible to run a 200Gb model, by using 20Gb of memory?
Size reduction is mostly based on Experts size. And it is limited by SSD speed. Check for Colibri and Flash-Moe, they are doing similar things with bigger models, but tok/s is not high