Given these numbers it has some potential to become quite usable for unattended workloads, especially if decode can be batched across multiple sessions (ideally enough of them to get some reuse of the sparsely streamed weights). (Of course this ultimately makes prefill times explode as you try and increase the workload even further. But that's arguably the natural bottleneck on any interesting local LLM inference, being a compute bound step.)
Given these numbers it has some potential to become quite usable for unattended workloads, especially if decode can be batched across multiple sessions (ideally enough of them to get some reuse of the sparsely streamed weights). (Of course this ultimately makes prefill times explode as you try and increase the workload even further. But that's arguably the natural bottleneck on any interesting local LLM inference, being a compute bound step.)
[dead]