I'm not the architect of journald and wasn't really around when these decisions were made, so I can't really speak authoritatively on that particular topic.
There was mailing list discussion at the time journald was conceived though, you can find it if you look.
It seems a recurring (handling) issue but unfortunately it affects multiple linux distro defaults. This is the worse that can collaboratively happen for FOSS in general imho.
systemd is not a collaborative project. It is Lennart's personal cathedral project and you can take it or leave it. That's fine for Lennart, the question is if it's so bad then why are the rest of us taking it instead of leaving it?
Probably because the overall impact is not as bad as extremely vocal people on GitHub and HN would have you believe, and more people like systemd than dislike it.
FWIW the journal file signature is "LPKSHHRH" for Lennart, Kay Sievers, Harald Hoyer, Red Hat... I presumed it was at least Lennart, Kay, and Harald who collaborated on the design.
> Performance: journal operations for appending and browsing should be fast in terms of complexity. O(log n) or better is highly advisable, in order to provide for organization-wide log monitoring with good performance
> Minimal Footprint: journal data files should be small in disk size, especially in the light that the amount of data generated might be substantially bigger than on classic syslog.
Sqlite3 is present in the default installation of most Linux distributions. It has proven itself from years of battle testing in many different environments. To use DuckDB or LevelDB would probably require pulling in an additional dependency.
No duckdb (or parquet). If you want to avoid writes and write amplification, you really want to avoid re-writing all 122880 rows of a row group every time a single insert happens.
Uh, who said anything about writing 122880 rows every time you do a single insert into DuckDB? There's a WAL. Consolidation happens in big chunks. (And it's not like journald log rotation is somehow better than WAL consolidation.)
We shouldn't be making momentus choices of data format based on vague and incorrect understandings of data formats.
WAL means it will write the same data at least twice. Similar issue, but even worse, with LevelDB -- it will just delay the inevitable huge rewrites for later. Funny to hear those proposals in the write amplification thread.
I believe journald log rotation is basically: close file - open a new one. How is it not completely different?
WAL writeback is at least principled and efficient. It works out to being equivalent to the custom Parquet-rotation things others mention, but already implemented and working.
So, yes, WAL for logs, because LSM is the design everyone converges on and a WAL is LSM. Better to use the LSM already implemented and debugged in a database than write some random new one in terms of Parquet that's going to have to do the same stuff in the end anyway, just with novel bugs and no tool support.
(And look, I don't give a damn what "DB" people say, a WAL writing back to a DB IS log... structured... merge under any fucking sensible definition of what LSM means.)
> journald does do a rewrite of the log file on rotation, so you're paying that IO anyway even if you ignore the dumb hash table updates.
You linked copy_file_atomic_at_full(), why? That function is not in the normal rotation path for journald, it's only used in a workaround when clearing FS_NOCOW_FL fails.
Rotation does not rewrite the log file normally, but there is a hole-punching operation though for reclaiming unused space.
So, yes, journald does in fact do bulk copies of log files on rotate. btrfs is hardly some fringe FS and its COW-flag behavior is documented and well-known. I'd expect extensive work on journald's storage engine to have uncovered this behavior at some point.
> When did this become a discussion limited to journald on btrfs?
btrfs is in the HN thread title.
> and that seems like something btrfs should fix at some point
Amazing. The Linux kernel should change to work around journald's inflexibility?
What someone should fix at some point is journald's strange IO patterns and hard-coded "helpful" attribute changes. I'd rather it just rename the file and let me do any defrag/compression/flag-setting I want than do anything with chattr behind my back in ways I can't even configure.
I'm not the architect of journald and wasn't really around when these decisions were made, so I can't really speak authoritatively on that particular topic.
There was mailing list discussion at the time journald was conceived though, you can find it if you look.
https://0pointer.de/blog/projects/the-journal.html might be a good entry-point.
The mailing list archives are here: https://lists.freedesktop.org/archives/systemd-devel/
It doesn't look like there was an open design review; Lennart Poettering just dropped it in in v38. https://lists.freedesktop.org/archives/systemd-devel/2012-Ja...
I think this also explains a lot and should not be ignored. https://github.com/systemd/systemd/issues/15292#issuecomment...
It seems a recurring (handling) issue but unfortunately it affects multiple linux distro defaults. This is the worse that can collaboratively happen for FOSS in general imho.
systemd is not a collaborative project. It is Lennart's personal cathedral project and you can take it or leave it. That's fine for Lennart, the question is if it's so bad then why are the rest of us taking it instead of leaving it?
Probably because the overall impact is not as bad as extremely vocal people on GitHub and HN would have you believe, and more people like systemd than dislike it.
[dead]
Ah, the Ulrich Drepper school of dealing with reported issues. Time for esystemd ;)
FWIW the journal file signature is "LPKSHHRH" for Lennart, Kay Sievers, Harald Hoyer, Red Hat... I presumed it was at least Lennart, Kay, and Harald who collaborated on the design.
... Sounds like a signature on a patch that triggers an epic Linus rant on LKML[1]
[1] Happened few times, I think RedHat as a whole even got banned from sending changes for a short while
Well there was an ambition, apparently.
> Performance: journal operations for appending and browsing should be fast in terms of complexity. O(log n) or better is highly advisable, in order to provide for organization-wide log monitoring with good performance
> Minimal Footprint: journal data files should be small in disk size, especially in the light that the amount of data generated might be substantially bigger than on classic syslog.
Because Poettering didn’t invent SQLite.
SQLite here is okay, but DuckDB or LevelDB would be better. Either way, no need to invent a new storage format.
Neither DuckDB nor LevelDB existed when journald was created. Not to say it couldn't be done today, but just some historical context.
LevelDB was released in 2011, so it existed but was very new.
Sqlite3 is present in the default installation of most Linux distributions. It has proven itself from years of battle testing in many different environments. To use DuckDB or LevelDB would probably require pulling in an additional dependency.
No duckdb (or parquet). If you want to avoid writes and write amplification, you really want to avoid re-writing all 122880 rows of a row group every time a single insert happens.
Uh, who said anything about writing 122880 rows every time you do a single insert into DuckDB? There's a WAL. Consolidation happens in big chunks. (And it's not like journald log rotation is somehow better than WAL consolidation.)
We shouldn't be making momentus choices of data format based on vague and incorrect understandings of data formats.
Write-Ahead Log for... logs?
WAL means it will write the same data at least twice. Similar issue, but even worse, with LevelDB -- it will just delay the inevitable huge rewrites for later. Funny to hear those proposals in the write amplification thread.
I believe journald log rotation is basically: close file - open a new one. How is it not completely different?
journald does do a rewrite of the log file on rotation, so you're paying that IO anyway even if you ignore the dumb hash table updates.
https://github.com/systemd/systemd/blob/8f4cd7de43d1e6e94687...
WAL writeback is at least principled and efficient. It works out to being equivalent to the custom Parquet-rotation things others mention, but already implemented and working.
So, yes, WAL for logs, because LSM is the design everyone converges on and a WAL is LSM. Better to use the LSM already implemented and debugged in a database than write some random new one in terms of Parquet that's going to have to do the same stuff in the end anyway, just with novel bugs and no tool support.
(And look, I don't give a damn what "DB" people say, a WAL writing back to a DB IS log... structured... merge under any fucking sensible definition of what LSM means.)
> journald does do a rewrite of the log file on rotation, so you're paying that IO anyway even if you ignore the dumb hash table updates.
You linked copy_file_atomic_at_full(), why? That function is not in the normal rotation path for journald, it's only used in a workaround when clearing FS_NOCOW_FL fails.
Rotation does not rewrite the log file normally, but there is a hole-punching operation though for reclaiming unused space.
> it's only used in a workaround when clearing FS_NOCOW_FL fails.
Clearing FS_NOCOW_FL doesn't work on btrfs for non-empty files. So what do you think journald is doing when it notices that it can't clear the flag?
When did this become a discussion limited to journald on btrfs?
and that seems like something btrfs should fix at some point
So, yes, journald does in fact do bulk copies of log files on rotate. btrfs is hardly some fringe FS and its COW-flag behavior is documented and well-known. I'd expect extensive work on journald's storage engine to have uncovered this behavior at some point.
> When did this become a discussion limited to journald on btrfs?
btrfs is in the HN thread title.
> and that seems like something btrfs should fix at some point
Amazing. The Linux kernel should change to work around journald's inflexibility?
What someone should fix at some point is journald's strange IO patterns and hard-coded "helpful" attribute changes. I'd rather it just rename the file and let me do any defrag/compression/flag-setting I want than do anything with chattr behind my back in ways I can't even configure.
> btrfs is in the HN thread title.
as is ext4
So write amplification on btrfs doesn't matter?