In the tracker, systemd opened issue #40262 on August 13 with a simple observation: a Debian 13 VM writes about 50 IOPS, while the log gets two lines per second. The author of the report considers the log format “extremely inefficient” and refers to a closed bug from four years ago without a solution. There are no replies from maintainers in the thread at the time of publication, and independent verification of the numbers is also missing, so treat them as a single measurement on one configuration.
But it’s a good reason: on people’s home servers this topic surfaces regularly and in a much more unpleasant form — “SD card died on Raspberry Pi in six months”, “journal ate 4 GB on VPS with a 20 GB disk”. Below are four typical symptoms, the mechanics behind each, and what really should be tinkered with. All default values come from man journald.conf.
Symptom 1. The disk is constantly writing something, even though there are almost no logs
What happens. The systemd journal is not a text file where a line is appended. It is an indexed database with hash tables and offset arrays, designed for fast journalctl -u nginx --since ... searches by any field.
When one line is written, the following are updated: the file header (record and object counters, tail offset, sequence number), for each unique pair “field=value” the DATA object (and systemd adds a dozen of them to your message: _PID, _UID, _SYSTEMD_UNIT, _BOOT_ID, _HOSTNAME, _MACHINE_ID, PRIORITY, and so on), new FIELD objects for new field names, chains in two hash tables, and ENTRY_ARRAYs.
On top of this, the write order is applied, precisely described in the format specification: first all new objects are appended to the end of the file, and only then are the references updated in the existing structures. In other words, the file is touched at least twice, in different places.
This is not a bug, but a conscious trade-off: the journal pays for writes to the disk so that field-based search is fast and does not require grep of gigabytes of text. The question is whether you need this trade-off on a device with limited write endurance.
What to do. If the machine is a home server where logs are needed “for flight analysis” rather than analytics, it makes sense to reduce what actually goes into the journal: remove debug levels from chatty units, and for containers switch Docker’s logging driver from journald to local or json-file with rotation — otherwise everything containers write will be duplicated in the system journal.
Symptom 2. The journal takes up gigabytes
What happens. By default SystemMaxUse is 10% of the filesystem size, but no more than 4 GB; SystemKeepFree is 15% free space; SystemMaxFileSize is one eighth of SystemMaxUse, but no more than 128 MB; SystemMaxFiles is 100 files. That is, on a 40 GB disk the journal legally can take 4 GB, and this is normal behavior, not a leak.
What to do. One-time cleanup:
journalctl --disk-usage
sudo journalctl --vacuum-size=200M
sudo journalctl --vacuum-time=14d
Constant limits — in /etc/systemd/journald.conf.d/00-limits.conf (a separate drop-in is better than editing the main file: it will survive package updates):
[Journal]
SystemMaxUse=200M
SystemMaxFileSize=20M
MaxRetentionSec=2week
After editing — sudo systemctl restart systemd-journald.
Symptom 3. SD card or cheap SSD degrades
What happens. This is where the mechanics from Symptom 1 turn into money. A constant stream of small writes to different parts of the file is the worst case for flash memory without a proper controller, and an SD card on a single-board computer is exactly such a case.
Migrating to /Storage=volatile blindly is not advisable: in this mode the journal lives only in /run and completely disappears after a reboot. If the server crashes at night, you won’t know why by morning.
Working scheme for single-board computers:
Storage=volatilein journald — the journal is in RAM, limited byRuntimeMaxUse;- simultaneously
ForwardToSyslog=yesand a lightweight syslog daemon that writes a plain text log with rotation — or immediately sends logs to another machine; - if there’s no external receiver — compromise: keep
Storage=persistent, but capSystemMaxUse=50Mand raiseSyncIntervalSecto reduce the number of disk flushes.
Symptom 4. After a hard reboot the journal is corrupted
What happens. SyncIntervalSec by default is 5 minutes. This is the timeout for forcing journal files to sync to disk. An exception is made for CRIT, ALERT and EMERG priorities: after such a message, the sync is performed immediately.
Two things follow. First, with sudden power loss you lose up to five minutes of logs — the very ones that are usually of interest. Second, this also leads to complaints about corrupted journal files: an unclosed file after a hard reboot has to be recovered.
Reducing SyncIntervalSec saves the logs, but increases the number of write operations — i.e., directly contradicts treating Symptom 3. You have to choose consciously: either the storage resource or the completeness of logs during a crash. A third option is to send logs to another machine and not solve this dilemma locally.
Two more settings people rarely remember
Compress is enabled by default, threshold is 512 bytes: anything larger is compressed. For typical single-line messages this means compression often does not trigger at all.
Seal (Forward Secure Sealing, HMAC-SHA-256 over the journal to protect against backdating) is described in the man page as enabled by default, but in practice you only get real sealing after you generate keys with journalctl --setup-keys. Without this step, TAG objects are not written to the file, and “enabled by default” does not actually mean it works.
What to check right now
journalctl --disk-usage # how much space is used
systemd-analyze cat-config systemd/journald.conf # which values are actually applied
journalctl --verify # is the journal intact
If --disk-usage shows hundreds of megabytes on a machine where you never read logs older than a week — that is the answer to where the disk resource went.
References
- systemd/systemd issue #40262 — report of excessive I/O in journald (numbers — from the author of the report, not independently confirmed)
- man journald.conf(5) — default values
- systemd.io — description of the journal file format
Who solved the logs issue on single-board computers: log2ram, moving logs to RAM, sending logs to a separate machine, or simply “installed an SSD and doesn’t think about it”? And has anyone encountered truly corrupted logs after power loss?