Flash next only needs 64GB for core model inference. If you really wanted it.
Need 64GB vram, 900+ GB/s speed, and a lot of system ram (128GB). Seems feasible. Hmm.
Maybe older GPUs work.
Flash next only needs 64GB for core model inference. If you really wanted it.
Need 64GB vram, 900+ GB/s speed, and a lot of system ram (128GB). Seems feasible. Hmm.
Maybe older GPUs work.
128GB on a Ryzen 395 is enough but not by a lot. Around 90GB for weights plus 10GB to 20GB for KV cache and checkpoints. I don't know about 64GB of VRAM for less than around $2500 by itself.
Tesla V100? Would have to be a 3-ish bit quant in 64GB ram