Can't you just look at memory bandwidth for generation?
You also need to look at memory size (to store the model). And processing can still be the bottleneck.
It’s more nuanced than that, but yes that’s the biggest factor.
You also need to look at memory size (to store the model). And processing can still be the bottleneck.
It’s more nuanced than that, but yes that’s the biggest factor.