Thanks! I think computers are fun and I want reading about them to be fun too.
I was also reminded of HazyResearch's MegaKernels. Didn't want to distract from the main thrust of the post, but definitely think that's a promising approach.
Thanks! I think computers are fun and I want reading about them to be fun too.
I was also reminded of HazyResearch's MegaKernels. Didn't want to distract from the main thrust of the post, but definitely think that's a promising approach.
There's some interesting work in NeurIPS this year on fused kernels for MoE too: https://flash-moe.github.io/