Complex systems have complex operations and problems. I think isolating the read and write workloads and separating the read queries by departments or even users is a distinct possibility especially with technologies like DuckDB.
Complex systems have complex operations and problems. I think isolating the read and write workloads and separating the read queries by departments or even users is a distinct possibility especially with technologies like DuckDB.
That is exactly where all the high scale stuff like Trino and StarRocks have gone (and ClickHouse can use S3 as storage too). However, the problem you run into is when you want to index stuff- secondary indices are tightly coupled to the query engine, so in practice you arent going to use StarRocks to write and index data to the object store, and then use ClickHouse or Trino to query it. I think it would be a useful development to decouple writes and reads, but this requires a common indexing scheme, and then once you have this standard, in principle lots more indexing plugins can be written and all compatible query engines can use them. Apache DataFusion is probably the right framework to build off of.