Use better databases? Learn prior art?

Dremel paper explained shredding in like 2010? Parquet had it in a while?

https://blog.x.com/engineering/en_us/a/2013/dremel-made-simp...

Yes, things like nested repeated fields (or structs) will mess you up hard time. When I was playing with Duckdb querying parquet several years ago I had issues with column names and even was digging some source code to improve statistics use, but it was doable and reasonably efficient on my datasets.

Although yes. If you have resources of large companies, both computational and manpower, you can do very nice things with your data. All that determination of “what would be nice to shred” based on real query statistics, compaction, regular optimization and improvements of data at rest by maintenance jobs… again, use better databases/engines and there’s no free lunch, you get what you paid for (hopefully).