Why not store everything in a single json column but dynamically generate index files, so finding rows is fast?
Then you never have to change the data. Only the index files.
Why not store everything in a single json column but dynamically generate index files, so finding rows is fast?
Then you never have to change the data. Only the index files.
This is a great question. The short answer is IO patterns: because the data is stored as Parquet in object storage just knowing that a certain page/row group/file has the data you are looking for is only half of the battle. If you then have to download MBs of JSON to then extract that one value from that one row things will still be slow. Shredding allows the data to be accessed independently, in addition to unlocking encoding (delta, ALP, etc.).
In practice we do both: we have key presence indexes and zone maps on shredded and unshredded values. Then we go an grab the data and if it's shredded it's going to be faster to grab even once we know where it is.
if you dynamically create indices of everything in the json, you've just doubled your storage space usage for nothing.