> What are you guys using it for?
Transparent queries on server/browser in identical sql.
If you play all the tricks Google's Capacitor does to the data layout, you can shove ~2-3 million rows to a browser based query engine in 16-20Mb with duckdb-wasm. We can scale it up to ~1Gb with HTTP range queries, with cache-control.
Then we use Jev to write SQL queries over it in 1-2s, whenever someone has a followup question. Getting Jev to write SQL takes a bit of prep, but my co-founder wrote a paper about clairvoyant query execution a few years ago [1].
I could have 3000 users logged into the same application, but all of them would be bringing their own CPU to run duckdb. My compute scaling problems for my data cube is a solved problem with this approach.
note that if you want range queries in duckdb-wasm to work, you currently need to make sure that you disable `builtin_httpfs`, otherwise you can end up pulling a ton of data: https://github.com/duckdb/duckdb-wasm/issues/2228#issuecomme...
This week I've been debugging a performance issue in a Ducklake and knocking out some good improvements, but this precise symptom—read_parquet() making tons of tiny range requests to load a parquet's footer from S3—was still baffling me. How lucky am I that I stumbled on your comment! Amazing, thank you for the pointer.
> read_parquet() making tons of tiny range requests to load a parquet's footer from S3
This is somewhat unavoidable if you have a number of files and a tight filter which rejects most of them. The footer choice was a bad one in hindsight (not for HDFS), but for anything which uses XOR or other kind of erasure coding. The other other file format I worked with used headers only instead of footers, to save a few ms in latency for the initial GET
I don't think Ducklake can maintain a zone-map or ART index over parquet files, but there is no equivalent of iceberg metadata/something which has min/max/HLL compacted across multiple files yet.
The secondary indexes with file identifiers stored in a sequential file is most likely the next useful thing duckdb can make.
Amazing we did the exact same thing at work. Been working great. A lot of BI datasets can fit within sub 20MiB of ZSTD compressed parquet.