It seems we could use a new kind of memory that streams the weight data in, like GDDR in reverse.
High Bandwidth Flash? https://www.sandisk.com/company/newsroom/blogs/2025/scaling-...
Optane? (Too soon?)
Optane was targeting the latency gap, while HBF targets bandwidth. Given how LLMs work, HBF is perfect for offloading.
interesting!
yes! I guess future hardware designs will have something like that!
High Bandwidth Flash? https://www.sandisk.com/company/newsroom/blogs/2025/scaling-...
Optane? (Too soon?)
Optane was targeting the latency gap, while HBF targets bandwidth. Given how LLMs work, HBF is perfect for offloading.
interesting!
yes! I guess future hardware designs will have something like that!