We're on the cusp of Kimi K3 becoming usable on sub-10k hardware.
https://github.com/gavamedia/deltafin
14 seconds per token? Not tokens per second. Seconds per token? That’s nowhere near the cusp!
For values of "usable" that include "14.6 seconds/token". It's a cool accomplishment! And newer hardware would speed it up some. But I think I'd want something a bit faster before declaring it usable in practice.
14 seconds per token? Not tokens per second. Seconds per token? That’s nowhere near the cusp!
For values of "usable" that include "14.6 seconds/token". It's a cool accomplishment! And newer hardware would speed it up some. But I think I'd want something a bit faster before declaring it usable in practice.