rsyslog: Root filesystem fills up

[filippocarletti]

The root filesystem unexpectedly fills up because log files (e.g., /mnt/data/log/messages) are written directly to the underlying root filesystem instead of the intended external/additional storage partition.

We suspect this is caused by a startup race condition between storage mounting (fstab) and logging initialization (rsyslog).

If the system suffers an unclean shutdown or crash, the kernel must perform journal recovery during the boot phase. This recovery delays the availability of the block device. Because S11fstab and S12rsyslog run almost back-to-back, rsyslog can start before the partition is fully mounted. By default, rsyslog implicitly creates missing directories, causing it to generate /mnt/data/log/ on the root filesystem and write to it.

Jun 15 22:29:10 firewall kernel: EXT4-fs (sda3): recovery complete
Jun 15 22:29:10 firewall kernel: EXT4-fs (sda3): mounted filesystem 59434584-fa25-4fe6-b3c5-e5ed46a52ebc r/w with ordered data mode. Quota mode: disabled.

---

Steps to reproduce

The issue is intermittent and typically occurs after a hard crash or power failure. To manually verify if logs are trapped underneath the mount point on the root FS, use the following steps:

# Stop logging and unmount the external storage
/etc/init.d/rsyslog stop
umount /mnt/data/

# Check the underlying root FS directory. If files exist here, the bug triggered.
# Clean them up to reclaim space:
rm /mnt/data/log/messages
rmdir /mnt/data/log

# Restore normal operation
block mount
/etc/init.d/rsyslog start

---

Proposed Fix

Prevent rsyslog from implicitly creating missing directories. If the external mount point is missing, rsyslog will resume writing to /mnt/data/log/messages as soon as the FS is mounted (some logs may be missing).

Enforcing $CreateDirs off in the global configuration ensures it will not generate directories on the root overlay.

diff --git a/packages/rsyslog/files/rsyslog.init b/packages/rsyslog/files/rsyslog.init
index 8ec71e50..2dd44ca0 100644
--- a/packages/rsyslog/files/rsyslog.init
+++ b/packages/rsyslog/files/rsyslog.init
@@ -97,6 +97,7 @@ expand_config() {
        echo "${input_t}" >> ${CONFIG_FILE}
        echo "${input_u}" >> ${CONFIG_FILE}
        echo "\$ActionFileDefaultTemplate ${template}" >> ${CONFIG_FILE}
+       echo "\$CreateDirs off" >> ${CONFIG_FILE}
        echo -e "${selectors}" >> ${CONFIG_FILE}
        echo -e "${forwarders}" >> ${CONFIG_FILE}
        echo -e "${includes}" >> ${CONFIG_FILE}

---

Note: in NethSecurity 8.8 victoria-metrics-data may have the same problem.