Yes, that's what I was saying w.r.t. expert offloading, i.e. ensuring that the GPU could fit the active parameters not all the parameters.
Yes, that's what I was saying w.r.t. expert offloading, i.e. ensuring that the GPU could fit the active parameters not all the parameters.
Alright, I guess I misunderstood. To be fair, this part:
> The Kimi-K2.6 model is 1.1T parameters with 32B active parameters. With light quantization (Q6_K) that's enough to run it (slowly) on a single 5090.
Is painting a very different perspective, even considering the latter parts it's hard to read that as "Of course offloading everything else that doesn't fit on the GPU itself". But anyways, it's been clarified now so no harm :)