> any model over about 128GB of weights isn't going to fit on the GPUs
Not sure if you misunderstand what GPUs we're talking about, one RTX Pro 6000 has 96GB of VRAM.
> Indeed. The M5 Ultra (currently up for pre-order) has 50% higher memory bandwidth than its predecessor, and a claimed 4x improvement in prompt prefill.
Exciting! Eagerly awaiting the benchmarks and comparisons then. Lets hope "4x improvement" had a good enough baseline so 4x actually ends up useful in practice compared to the current hardware they offer.
> 50% higher memory bandwidth
Seems this lands on ~1.2 TB/s if what Apple claims is correct. For reference, RTX Pro 6000 does 1.8 TB/s, so seems Apple is indeed getting closer incrementally.
> Not sure if you misunderstand what GPUs we're talking about, one RTX Pro 6000 has 96GB of VRAM.
Right, but the top option here is a pair of RTX Pro 6000s, hence 192 GB of VRAM in total. Should be enough for a 128GB model plus context, cache, etc.
Yeah, that's why the "any model over about 128GB of weights isn't going to fit on the GPUs" part doesn't make sense, you'll easily be able to run weights in that weight class on two of them. Or did I misunderstand what "on the GPUs" you meant?