SQLite has the problem of a malicious reader being able to hold up writers. Maybe that's fine in most cases, but in a system log, I don't think that's acceptable. IMHO, options are to indirect through a daemon anyway (e.g. using Quack) or do a lot of engineering to make it possible for an unprivileged reader to open() the log file and read it in such a way that it can't interfere with privileged writers.
Your LSM compaction strategy is going to have to solve the same problem anyway, isn't it? DuckDB is an LSM compaction strategy of this form, already done.
I don’t think there’s any hard work here. Other than the append-only part, all files would be either immutable (the Parquet parts) or maybe mutated by wholesale atomic replacement of the inode (the catalog, although the directory itself, via its contained filenames) could maybe do that. Readers might have to retry sometimes, but readers would neither have boy expect any write privileges.
How do readers get log entries that haven't made their way into one of the parquet archive files? If the tip is some kind of live-update DB, that DB has to support concurrent readers who can't block writers. Or would you just make log messages invisible to readers until they made their way into a stable Parquet file?
Forget about DB terminology and look at what's happening ON THE DISK. ON THE DISK, is what DuckDB doing any less efficient than what your custom Parquet thing would be doing?
On the disk the live update part would be either a circular buffer (and readers would need to double-check the start/end marks after reading) or just literal append-only streams. In the latter case, reading would be barely more complex than tail -f.
Sure, but you have to take pains to make sure that journalctl -f doesn't skip events or print some twice. It can be made to work, for sure, but it just seems easier to print logs via a daemon instead, especially because if you go through a deamon, you turn the disk format from an interface into an implementation detail.
I want my logs readable on a completely read-only disk. Sure, I can start a daemon, but that’s mildly annoying.
Also, the syslog daemon should be extremely reliable, and throwing giant table scans at it makes this more complex.