Nice introduction to a simple but useful idea! The Lagrangian works like a time-smoothed optimizing direction state, but it can be placed on any wire, even at non-differentiable boundary! Can it be better than existing training methods for discrete components like argmax, MoE or VQ-VAE? Maybe networks can be composed by a lot of learnable discrete components, or even bits and gates finally.
other questions:
- can it be used to relax timing order requirement in pipeline parallelism? Each node update lambda on communication, and optimize weights at other time. - given that BP is using SGD, can batches and T share the same timeline in optimization, while keeping the descent direction?