Any recommendations from HN for a write-once (literally once), data storage format that's suitable for network storage?
sqlite docs recommend avoiding using it on network storage, though from what I can gather, it's less of an issue if you're truly only doing reads (meaning I could create it locally and then copy it to network storage). Apache Parquet seems promising, and it seems to support indexing now which is an important requirement.
SQLite works fine over read-only NFS, in my experience. Just only work on an immutable copy and restart your application if ever changing it. If your application is short lived and can only ever see an immutable copy on the path, then it is a great solution.
Ordinary files inside squashfs?
https://www.kernel.org/doc/html/latest/filesystems/squashfs....
Multiple writers on network storage is the issue. Reading should be totally fine.
SQLite does work on NFS even in read-write scenario. Discovered by accident, but my statement still holds. The WAL mode is explicitly not supported over network filesystems, but I guess you don't expect it to :)
My experience has been the opposite... Lots of db lock and corruption issues. The FAQ doesn't call out WAL specifically, just says don't do it at all: https://www.sqlite.org/faq.html#q5
I've had multiple flaky issues with SQLite (e.g. non-HA Grafana) on Azure Files using NFS v4.1 leading to locked DBs. Perhaps some implementations work, I'm not gonna rely on it or advise others to do so.
Yeah trying to write from several hosts will certainly fail if you don't have advisory locks working, which is not a given, so you are right of course
These were singular containers, let alone hosts.
Parquet files are what I use.
SQLite over NFS works if you have one writer and many readers.