Author here. Very cool to see you commenting here, ClickHouse is such an amazing system!
But yes, what's old is new. I didn't mean to claim we invented this technique. I actually cited ClickHouse's article (https://clickhouse.com/blog/a-new-powerful-json-data-type-fo... in the article) in the post :
> Our shredded data layout predates both the Parquet Variant type and ClickHouse's JSON support but is very similar to these approaches.
I said _predates_ based on when the article I cited came out, but based on your linked PR it sounds you landed it earlier than I thought, I'll remove the wording. I'm not sure where we originally got the idea from or where it comes from, I recall hearing the Databricks and BigQuery folks sharing similar ideas around the same time. Do you recall where the idea originated / where did you first see it? Happy to include that in the article!
Most of our work went into making the technique work w/ Parquet in object storage queried with DataFusion. We had to rewrite big chunks of the Parquet scanning machinery, filter pushdown optimizers, etc.
Long term we're going to move a lot of this to the now "standard" Variant data type, which we are partially responsible for bringing into the Rust / DataFusion ecosystem.