What are the new file format initiatives you're referencing here?
This solution seems clever overall, and finding a way to bolt on features of the latest-and-greatest new hotness without breaking backwards compatibility is a testament to the DataFusion team. Supporting legacy systems is crucial work, even if things need a ground-up rewrite periodically.
Off the top of my head:
- Vortex https://github.com/vortex-data/vortex
- Lance https://github.com/lancedb/lance
- Nimble https://github.com/facebookincubator/nimble
There are also a bunch of ideas coming out of academia, but I don't know how many of them have a sustained effort behind them and not just a couple of papers
Lance (from LanceDB folks), Nimble (from Meta folks, formerly known as Alpha); I think there are a few others
https://github.com/lancedb/lance
https://github.com/facebookincubator/nimble