It doesn't need extra quantization. The official weights are natively mixed precision FP4/FP8, so it fits in ~160GB. The API slowness is probably from being batched with other concurrent user requests. The provider's aggregate throughput gets higher but per-stream speed slows down.
You are right, relatively to other llm providers this is not slow. But if you think what is possible when you have 1000t/s a sec you might find it slow.
It doesn't need extra quantization. The official weights are natively mixed precision FP4/FP8, so it fits in ~160GB. The API slowness is probably from being batched with other concurrent user requests. The provider's aggregate throughput gets higher but per-stream speed slows down.
V4 Flash fits entirely in two RTX Pro 6000s without any quantization at all.
From opencode go $10/mo plan I get between 60 t/s and 100 token/s even with large contexts of 150k+ tokens.
I wouldn't call 80 t/s slow.
You are right, relatively to other llm providers this is not slow. But if you think what is possible when you have 1000t/s a sec you might find it slow.
That's across 64 concurrent streams; you could make more concurrent requests to DeepSeek API no?