Hi, author here.

The article doesn't claim to have made some huge discovery, it tells the journey of making our product / system much better by applying well known techniques. We linked to similar things (ClickHouse's handling of semi structured data, Parquet's Variant type). There are others (e.g. BigQuery). I'm sure there's some system that did similar things before (if you have a reference I'd be interested in learning the history here) but all of the ones I mentioned above are from 2022+.

But besides that I think your comment misses the entire point of the article: how do you normalize semi-structured data that varies in schema vastly across tenants, within those tenants between those systems and drifts over time rather rapidly? There is no schema to pick and normalize to.

Hi, author here.

I agree this is not some huge discovery. We link ClickHouse's JSON type and Parquet Variant in the post, and Snowflake's VARIANT and BigQuery do similar things. We didn't mean to present it as a discovery. The post is about how we applied it to our system.

The part normalization doesn't answer for us, as guites said in this thread, is that there's no schema to normalize to. Every tenant sends different OpenTelemetry attributes, different services within a tenant send different ones, and they change from week to week. Picking a fixed set of columns is what we tried first (we call it static shredding in the post). So each Parquet file now picks its own columns from the data it contains, without any configuration.

I know in the database world we are often recycling or tweaking ideas from decades ago so if you know where these ideas for handling semi-structured data in columnar databases comes from I'd be interested to learn!