Their preprint showed it worked well with small batch size, but most prod deployments right now have rather large batch sizes from serving multiple users
Their preprint showed it worked well with small batch size, but most prod deployments right now have rather large batch sizes from serving multiple users
That's a great answer, thanks. I'd missed that detail.
Honestly, a lot of those issues could be remedied with more optimized kernels, which LLMs could definitely write quite well now. I wouldn't be surprised if some deployments of Gemini Flash switched to diffusion for cases where there is low utilization, since Google showed you don't need to retrain an autoregressive model to rework it for diffusion inference.
> since Google showed you don't need to retrain an autoregressive model to rework it for diffusion inference
IIRC you do need to retrain the model, just closer to the level of finetuning rather than e.g. pretraining from scratch. Which also means that the model will perform differently with a different capability profile.
If that's true, that's the real answer, or more like anty-answer because you're saying basically: hey, nothing should stop them.
Google's gonna Google...
I wonder if it'd work better as a local model