Add remote syslog forwarding for deployed nodes - #236
Open
Obihoernchen wants to merge 9 commits into
Open
Conversation
5 tasks
Member
Author
|
Thoughts:
|
Member
Author
|
On a 16 CPU VM on EPYC I get:
|
Obihoernchen
marked this pull request as draft
July 15, 2026 17:59
A new `logging.servers` attribute, when set, makes OS deployment configure the node to forward its logs to the given servers. It is surfaced as loggingservers/loggingmethod in confluent.deploycfg and applied by the new common `setuplogging` script, which is invoked in the profiles of all OSes shipping rsyslog by default and does nothing when the attribute is unset or the selected forwarder is absent. The forwarding method is selected by `logging.method`: `rsyslog` (the default) writes an rsyslog drop-in forwarding syslog to the given servers over UDP port 514; `journal-remote` configures systemd-journal-upload to upload the journal to systemd-journal-remote on port 19532 (a single destination only - the first server is used). A `logging.tls` attribute is defined to encrypt the forwarding with TLS authenticated via the confluent certificate authority (not implemented yet) Example receiver configurations for the confluent server side (rsyslog per-node logs plus logrotate, journal-remote drop-in) are shipped under `/opt/confluent/share/examples/logging`.
When the logging.tls attribute is set, the log forwarding configured from logging.servers is now encrypted and mutually authenticated via the confluent TLS certificate authority. During deployment, setuplogging requests a node certificate from the deployment API (/confluent-api/self/tlscert, validity governed by pubkeys.tls_lifetime, 47 days unless raised) and trusts /etc/confluent/ca.pem. With the rsyslog method, logs are forwarded over TCP port 6514 using the rsyslog-openssl or rsyslog-gnutls stream driver with x509/certvalid authentication; with journal-remote, systemd-journal-upload uploads over HTTPS port 19532 with explicit key, certificate and trust anchor settings. If TLS is requested but prerequisites are missing, the node is left unconfigured. The certificate material is stored in dedicated directories readable by the consuming service (/etc/rsyslog.d/confluent-tls for rsyslog, /etc/ssl/confluent-logging for the journal tools). The receiving side is handled by a new confluent-logging-receiver-setup helper shipped with the examples under /opt/confluent/share/examples/logging. It issues a server certificate signed offline by the local confluent CA (SANs covering the server names and addresses), installs the mutual TLS reception configuration for either method, setting TrustedCertificateFile explicitly for systemd-journal-remote, and can alternatively install the plaintext example configurations. Note: Some distros (e.g., EL10, Ubuntu 24.04) build systemd-journal-remote without GnuTLS. This silently disables client certificate verification (accepting any client), though encryption remains active. The helper warns of this issue. If client verification is required, use rsyslog (as noted in logging.tls).
The per-node log files on the receiving server were named after the client-supplied syslog HOSTNAME, which any authenticated sender can forge, allowing one node to inject entries into another node's log file. Name the files by FROMHOST instead (the sender identity as resolved by the receiving server from the peer address) so injected entries always land under the actual sender.
The log transports offer no certificate revocation, so note in the logging.tls attribute description that pubkeys.tls_lifetime should be chosen with node decommissioning in mind when raising it.
The confluent CA issues certificates to every node and BMC, all technically usable for TLS server authentication, so validating only the certificate chain allows any of them to pose as the logging server towards forwarding nodes if traffic can be redirected. When a logging.servers entry is a DNS name, forward with x509/name authentication pinning that name as the only permitted peer. The receiver certificates issued by confluent-logging-receiver-setup already carry the server names as DNS-type subject alternative names. rsyslog matches pinned peers only against DNS-type names, so address-valued entries keep chain validation (x509/certvalid); the logging.servers documentation now recommends names for this reason. The journal-remote method needs no distinction, as systemd-journal-upload verifies the URL host against the certificate either way.
The TLS forwarding actions run in rsyslog's main queue by default, so a log server that stops reading (as opposed to one that is down, which merely suspends the action) blocks the omfwd send and with it every other action on the node, including local log files. Give each forwarding action a bounded in-memory queue so an unreachable or hung receiver costs at most the queued messages: local logging continues, messages buffer across shorter outages and are dropped only once the queue fills. The non-TLS UDP method needs no queue, as it cannot block.
The minimal receiver examples run into two rsyslog defaults well below the cluster sizes confluent deploys: imtcp refuses more than 200 concurrent connections, so most of a larger cluster of TLS forwarders (one persistent connection each) cannot connect at all, and omfile caches only 10 dynafile handles, reopening a file for nearly every message once logs from many nodes interleave. Size both for clusters on the order of 4k nodes, reap dead sessions with TCP keepalive, absorb bursts (whole racks booting, reconnect storms after a receiver restart) with more UDP receiver threads, a larger receive buffer and a dedicated queue that also keeps node traffic and the server's own logging from delaying each other, and batch the file writing asynchronously. All parameters are accepted by the oldest rsyslog among the supported distributions (8.2312 in Ubuntu 24.04); imtcp workerthreads is not, so the receiver setup helper probes for it and enables it where available rather than placing it in the example.
The forwarding certificate was issued once during deployment, so with the default 47 day pubkeys.tls_lifetime the log forwarding silently stopped when it expired, and the documented workaround of a multi-year lifetime directly worsened the lack of revocation in the log transports. Have setuplogging install a renewal script and a daily randomized systemd timer that requests a fresh certificate through the deployment API once less than half of the validity remains. A failed renewal never disturbs the material in use: the replacement is staged beside the live files and only swapped in, and the consuming service only restarted, on success.
The plaintext log-forwarding path used UDP (imudp receiver, omfwd protocol="udp"), which silently drops messages once a burst outruns the receiver's socket buffer -- exactly the synchronized bursts a large cluster produces (a whole rack booting, a reconnect storm after the receiver restarts). Switch it to plain TCP: imptcp on the receiver and protocol="tcp" on the node side. TCP back-pressures the senders instead of dropping, so bursts are absorbed rather than lost, and imptcp (unlike imtcp) imposes no session limit, so every node can hold a persistent connection up to the receiver's open-file limit. The node side gains the same bounded async retry queue the TLS path already uses, since a TCP action can block where UDP could not. Measured on ~4k simulated senders, imptcp delivered every message losslessly where imudp shed 50-75% under the same overload, for the cost of one persistent connection per node.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements xCATs
syslogpostscript functionality. But more advanced.New node attributes
Adds
logging.servers,logging.methodandlogging.tlsnode attributes. During deployment the node is configured to forward its logs via rsyslog (default) or systemd-journal-upload, optionally TLS-encrypted and mutually authenticated through the confluent CA.Architecture
/var/log/confluent/nodes/<node>.log, named by the receiver's view of the sender, not the client-supplied hostname or systemd-journal-remote:/var/log/journal/remote/remote-<nodeip>.journal(filename hardcoded in systemd-journal-remote).confluent-logging-receiver-setuphelper + example receiver configs, tuned for up to 4K nodespubkeys.tls_lifetimejust works and short lifetimes double as the revocation substitutersyslog vs systemd-journal-remote
Overall rsyslog can handle ~x10 times the messages/s systemd-journal-remote can. systemd-journal-remote has no tuning options and was not built for scale. Furthermore, some OSs can't verify the client TLS cert (see below). But there is another big benefit in using it: The additional journalctl metadata you can use with commands like
journalctl _HOSTNAME=node001 -u sshd.service -D /var/log/journal/remotesystemd-journal-remote TLS client certificate verification
systemd-journal-remoteonly verifies client certificates when systemd is built with gnutls; otherwiseTrustedCertificateFile=is silently ignored. The transport is still encrypted and the node still verifies the server, but uploads from any client are accepted. The setup helper detects this and warns; use the rsyslog method where client verification is required.OS support for systemd-journal-remote TLS client certificate verification looks like this:
rsyslog transports
Plain: TCP/514 via
imptcp. Core modules only (imptcp/omfwd, nothing extra to install on nodes), andimptcpimposes no session cap, so every node holds one persistent connection (bounded only by the receiver's open-file limit). Unlike UDP, TCP back-pressures senders under load instead of dropping, and a bounded client-side queue spools during outages. Under a simulated ~4K-sender burst the difference is:Tested on 8 core receiver VM, 32 core sender VM:
imudp(UDP)imptcp(TCP)UDP has no backpressure, so a synchronized burst (a rack booting, a reconnect storm after the receiver restarts) simply overruns the receiver's socket buffer; imptcp delivered every message losslessly and scales to 4K connections cheaply.
TLS: TCP/6514 via
imtcp, openssl driver with gnutls fallback. The standard syslog-over-TLS transport; both driver modules are packaged on all supported distributions, andx509/certvalid/x509/namemap directly onto the confluent CA model (accept any node certificate from the CA, pin the server identity).RELP was rejected: an extra package on every node, and its only mutual-auth modes (fingerprint/name pinning) cannot express CA-chain trust. This is unmanageable with short-lived, per-deployment node certificates; its delivery acks also gain little while journald rate-limiting drops messages upstream anyway.
DTLS requires rsyslog ≥ 8.2402 (Ubuntu 24.04 ships 8.2312).