No duckdb (or parquet). If you want to avoid writes and write amplification, you really want to avoid re-writing all 122880 rows of a row group every time a single insert happens.
No duckdb (or parquet). If you want to avoid writes and write amplification, you really want to avoid re-writing all 122880 rows of a row group every time a single insert happens.
Uh, who said anything about writing 122880 rows every time you do a single insert into DuckDB? There's a WAL. Consolidation happens in big chunks. (And it's not like journald log rotation is somehow better than WAL consolidation.)
We shouldn't be making momentus choices of data format based on vague and incorrect understandings of data formats.
Write-Ahead Log for... logs?
WAL means it will write the same data at least twice. Similar issue, but even worse, with LevelDB -- it will just delay the inevitable huge rewrites for later. Funny to hear those proposals in the write amplification thread.
I believe journald log rotation is basically: close file - open a new one. How is it not completely different?
journald does do a rewrite of the log file on rotation, so you're paying that IO anyway even if you ignore the dumb hash table updates.
https://github.com/systemd/systemd/blob/8f4cd7de43d1e6e94687...
WAL writeback is at least principled and efficient. It works out to being equivalent to the custom Parquet-rotation things others mention, but already implemented and working.
So, yes, WAL for logs, because LSM is the design everyone converges on and a WAL is LSM. Better to use the LSM already implemented and debugged in a database than write some random new one in terms of Parquet that's going to have to do the same stuff in the end anyway, just with novel bugs and no tool support.
(And look, I don't give a damn what "DB" people say, a WAL writing back to a DB IS log... structured... merge under any fucking sensible definition of what LSM means.)
> journald does do a rewrite of the log file on rotation, so you're paying that IO anyway even if you ignore the dumb hash table updates.
You linked copy_file_atomic_at_full(), why? That function is not in the normal rotation path for journald, it's only used in a workaround when clearing FS_NOCOW_FL fails.
Rotation does not rewrite the log file normally, but there is a hole-punching operation though for reclaiming unused space.
> it's only used in a workaround when clearing FS_NOCOW_FL fails.
Clearing FS_NOCOW_FL doesn't work on btrfs for non-empty files. So what do you think journald is doing when it notices that it can't clear the flag?
When did this become a discussion limited to journald on btrfs?
and that seems like something btrfs should fix at some point
So, yes, journald does in fact do bulk copies of log files on rotate. btrfs is hardly some fringe FS and its COW-flag behavior is documented and well-known. I'd expect extensive work on journald's storage engine to have uncovered this behavior at some point.
> When did this become a discussion limited to journald on btrfs?
btrfs is in the HN thread title.
> and that seems like something btrfs should fix at some point
Amazing. The Linux kernel should change to work around journald's inflexibility?
What someone should fix at some point is journald's strange IO patterns and hard-coded "helpful" attribute changes. I'd rather it just rename the file and let me do any defrag/compression/flag-setting I want than do anything with chattr behind my back in ways I can't even configure.
> btrfs is in the HN thread title.
as is ext4
So write amplification on btrfs doesn't matter?