This is quite an old technique. In ClickHouse, it appeared before it was named "shredding": https://github.com/ClickHouse/ClickHouse/issues/54864
This is quite an old technique. In ClickHouse, it appeared before it was named "shredding": https://github.com/ClickHouse/ClickHouse/issues/54864
On the flipside, I just submitted https://news.ycombinator.com/item?id=50187394 which it looks like ClikcHouse actually copied (with attribution) in https://github.com/ClickHouse/ClickHouse/pull/119230, very cool to see ideas percolating like this!
Author here. Very cool to see you commenting here, ClickHouse is such an amazing system!
But yes, what's old is new. I didn't mean to claim we invented this technique. I actually cited ClickHouse's article (https://clickhouse.com/blog/a-new-powerful-json-data-type-fo... in the article) in the post :
> Our shredded data layout predates both the Parquet Variant type and ClickHouse's JSON support but is very similar to these approaches.
I said _predates_ based on when the article I cited came out, but based on your linked PR it sounds you landed it earlier than I thought, I'll remove the wording. I'm not sure where we originally got the idea from or where it comes from, I recall hearing the Databricks and BigQuery folks sharing similar ideas around the same time. Do you recall where the idea originated / where did you first see it? Happy to include that in the article!
Most of our work went into making the technique work w/ Parquet in object storage queried with DataFusion. We had to rewrite big chunks of the Parquet scanning machinery, filter pushdown optimizers, etc.
Long term we're going to move a lot of this to the now "standard" Variant data type, which we are partially responsible for bringing into the Rust / DataFusion ecosystem.