Nice job implementing expert caching!
Thank you! Under good conditions it achieves approx a 67% cache hit rate with 16 expert slots
That's great, now I wonder how cache hit rate scales for larger models. Do you have any plans trying Qwen 3.6 or larger?
Thank you! Under good conditions it achieves approx a 67% cache hit rate with 16 expert slots
That's great, now I wonder how cache hit rate scales for larger models. Do you have any plans trying Qwen 3.6 or larger?