journald is IMO the worst part of the systemd ecosystem. You're better off using it only as a router and not storing any logs in it. The indexing system it uses is slow and provides no control over chatty subsystems - you cannot truncate the logs for just a single identifier. For all the use indexing is doing you will get better performance out of a modern grep like ag or rg. Structure is worth something but it's better off somewhere other than journald.
I would much rather that they had used an existing database file format. Sqlite3 is robust and already present in the default installation of most Linux distributions. Querying system logs with SQL would be cool and likely faster than using the sd_journal API with all it's weird quirks.
If you want to store logs in a database, just use standard rsyslog. It has supported database backends pretty much since its inception at the dawn of the century. No need to reinvent anything.
> No need to reinvent anything.
Well I think we found the reason for journald's complexity right there, systemd devs and reinventing the wheel (plus breaking backwards compat in the process) is a match made in heaven
Text or text-like (e.g. text content with simple control char delimiters for metadata) would be far superior than the slow-down from Sqlite's safety mechanisms. Optimising logs for read, at the expense of write, is a bad pattern to me.
Read optimized? That is funny because reading logs from journald is dog slow compared to, you know, log files.
I was talking about alternatives like Sqlite. It might optimise complex querying, but writing to it is slower than simple appends.
There's an open source project which is a syslog daemon that stores logs in DuckDB. You can configure journald to forward logs to it.
https://github.com/phare/sloggo
I recently put a lot of effort into reducing logging because of excessive writes. It was so much easier when everything had its own log and you could just look at which files were growing.
Systemd is touted as being highly modular. So it should be easy enough to replace the logging module journald.
Why hasn't this been done if it's that terrible?
While the rest of systemd is actually pretty modular, journald is unfortunately the only other required component. You can not run systemd without journald running in some way; closest you can get is setting Storage=none and forwarding the logs elsewhere.
journald is in a weird state where its "good enough" and mandatory that most people forget how bad it is until something like this pops up.
Normally I'm pretty happy with systemd and its many components; I even willingly run systemd-resolved, which is probably the other most hated component. But journald makes a lot of weird choices and if I could drop it I would in a heartbeat.
Because it's a big fat lie
You can trivially configure it to forward logs to another service to manage them.
that's not the point, why ship most common linux distros with an unreliable logging solution by default?
because its everything or nothing with systemd its a monolith wearing a trenchcoat