Fair point. If only the paper had a link to the code.
I found this[1] repo which tries to reproduce it, and its accompanying blog post[2] uses (z_t, z_t - delta), without further mention.
Another repo[3] trying to reproduce it also does the reverse from what I can see, so (current, past).
Interestingly the first repo got underwhelming results, while the second repo matched the paper roughly. Not sure if it's related to the Claude-usage or not in the first, though it notes a lot of details were left out of the paper which is true.
[1]: https://github.com/Infatoshi/grassmann-flows/
[2]: https://github.com/Infatoshi/grassmann-flows/blob/main/blog....
They advertised it as 10% to 15% worse than transformers
Not sure what the point of the paper's title is if the resulting model does not match the transformer.
Well they're quite explicit about that:
In summary, our results suggest that what we fundamentally need for strong sequence modeling is not attention as such, but a principled way for representations to move on the manifolds they inhabit. Grassmann flows provide one concrete instantiation of this idea, and we hope they will encourage further exploration of geometric alternatives to attention in the design of neural architectures.
They also note they kept the proposed model very simple to make it easier to understand (and presumably train):
In this paper we deliberately restricted ourselves to a minimal design—k = 2, no explicit global invariants—to keep the core idea clear. But we view “global invariants + local Grassmann flow” as a promising direction for future work on geometry-aware reasoning.