See the recent development of DiLoCo at Nous Research and Prime Intellect.
oh that's very cool, exactly the kind of stuff we need
oh that's very cool, exactly the kind of stuff we need