Seems like a good way to shorten the lifespan of an NVME SSD significantly by using up its TB written lifespan, if you let it extensively swap. Also the performance will be absolutely abysmal like 0.1 tok/second.
Seems like a good way to shorten the lifespan of an NVME SSD significantly by using up its TB written lifespan, if you let it extensively swap. Also the performance will be absolutely abysmal like 0.1 tok/second.
The LLM tools are smart enough to keep the weights on the disk and read-write stuff in RAM.