Consumer inference scenarios tend to be highly bespoke so it's difficult to apply a monokernel approach based on deep manual optimization. I suppose this could become applicable to rare scenarios where both the model and the hardware are fixed and self-contained, e.g. I'm running Apple's AI model on the latest Apple Silicon hardware. Then this becomes a viable approach even for 'consumer' use.

The authors' approach also encompasses multi-node approaches that won't apply easily to consumer inference since consumer GPUs have very low-performance interconnects, hence why layer parallelism is usually favored. (But that doesn't work very well with the monokernel approach, since it involves running distinct logic on each separate GPU. It also doesn't speed up single inference, though you can get that throughput back by pipelining small minibatches.)