Just curious, what is your ram and how many tokens per second do you get with that?
RAM 32GB, tokens are less than 20, may be about 12-15. Main issue is context window, I have set to 64k but practically beyond 32k, too much swap and slow down.
I am getting like ~25 tps with M5 Max 64GB. Unsloth Desktop and Qwen 3.8 27B Q6 KXL
RAM 32GB, tokens are less than 20, may be about 12-15. Main issue is context window, I have set to 64k but practically beyond 32k, too much swap and slow down.
I am getting like ~25 tps with M5 Max 64GB. Unsloth Desktop and Qwen 3.8 27B Q6 KXL