I wish we had some kind of distributed training capability... Like Folding@home, but for LLMs.

See the recent advance of DiLoCo at Nous Research and Prime Intellect.