lt-diode-pro

Hardened, streaming LT (Luby Transform) fountain-code transport for one-way data diodes.

Send files reliably over a strictly one-way channel (a hardware data diode, or any lossy UDP link with no feedback and no retransmission). Data is encoded with rateless erasure coding (LT codes over the Robust Soliton Distribution): the sender emits a stream of XOR-combined coding packets, and the receiver reconstructs the file from any sufficiently large subset that arrives — it never asks for a packet to be resent, because on a diode there is no way to ask.

This is a production-grade fork of Petra's melorian94/robust-soliton-LT-C: multi-terabyte scale, OpenMP-parallel + batched I/O at line rate, zero-temp streaming (pipe zfs send straight in and zfs receive straight out), a built-in checksum gate, FILEID path-traversal hardening, and an optional --pipe output mode. See CHANGELOG.md for the full diff vs. upstream. Produced at Politehnica University Timisoara.

Features

• No feedback / no retransmission — correct by design for true data diodes.
• Multi-terabyte transfers (64-bit offsets; MAX_GROUPS ≈ 9.5 TiB). Validated end-to-end on a continuous 1 TiB transfer at loss margin 30 (~4 h 41 min, ~525 Mbps goodput, 0 decode failures, whole-file checksum match, 0 queue drops).
• Parallel multi-file (file/daemon mode): up to MAX_INFLIGHT (default 8) transfers with distinct FILEIDs decode concurrently, each routed to its own context and finalized independently. The single-file fast path is unchanged. (--pipe is single-file — see below.)
• Line-rate on 1 GbE: OpenMP-parallel encode/decode, sendmmsg/recvmmsg, no per-packet pacing.
• Zero temp space on both ends: the sender reads stdin, the receiver decodes on the fly with bounded RAM (does not grow with file size) and writes only the 1× output.
• Integrity gate: whole-file checksum verified on the receiver; corrupt transfers are never promoted.
• --pipe mode: decoded output straight to stdout (single-shot, zcat-like) for zfs receive.

Build (CMake)

Binaries are CPU-specific by default (-march=native) — build on (or for) each target host.

cmake -S . -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build -j
# binaries: build/sender_stream  build/receiver_stream

Useful options: -DLTPRO_NATIVE=OFF (portable build), -DMAX_GROUPS=N (receiver per-transfer cap), -DMAX_INFLIGHT=N (receiver: default for max parallel transfers, 8 — now runtime-overridable via the max_inflight config key / --max-inflight, clamped to [1, MAX_INFLIGHT_CAP]), -DMAX_INFLIGHT_CAP=N (receiver: hard ceiling on the runtime depth, default 256), -DEVICT_TIMEOUT_S=N (receiver: default seconds before a stalled transfer with a lost EOT is evicted as corrupt, 60 — now runtime-overridable via evict_timeout / --evict-timeout; use a small value in tests), -DSEND_PACE_US=20 (throttle the sender for a loopback test). Requires a C11 compiler with OpenMP.

Usage

The sender reads the payload on stdin; SEED and loss% must be identical on both ends. The codec uses three consecutive UDP ports: port, port+1, port+2.

# receiver (file mode, daemon): writes <temp>/<name> + <name>.finished when the checksum matches
receiver_stream [--pipe] [--progress] [--syslog | --log <file>] [--log-level <lvl>] \
                [--max-inflight <n>] [--evict-timeout <s>] \
                [--config <file>] [<port> <temp-folder> <SEED> <loss%>]

# sender: stream any data in on stdin
cat bigfile | sender_stream [--progress] [--syslog | --log <file>] [--log-level <lvl>] \
                            <DST-IP> <port> <SEED> <name> <loss%>

# end-to-end ZFS replication, zero temp on both ends:
zfs send tank/ds@snap | sender_stream 10.0.0.3 9000 42 repl 30        # sender host
receiver_stream --pipe 9000 /tmp/ltc 42 30 | zfs receive tank/ds      # receiver host (use: set -o pipefail)

• loss% is the redundancy margin (e.g. 30). Higher = more resilient to loss but lower goodput. On a one-way link there is no resend, so choose conservatively for your link quality.
• --pipe makes the receiver single-shot and single-file: it streams the decoded bytes of the first FILEID to stdout in order, then exits with the checksum verdict as its exit code (0 = OK, non-zero = corrupt). A pipe is one stream, so any additional FILEID seen in --pipe mode is rejected with a warning on stderr. Without --pipe the receiver is a file-mode daemon (out-of-order writes, .finished/.corrupt gate, loops forever) and decodes multiple files in parallel (see below).

File-mode daemon: parallel multi-file, no-overwrite naming, eviction

• Parallel transfers. Each incoming FILEID is routed to its own decode context; up to MAX_INFLIGHT (default 8) files decode at once. A new FILEID arriving while 8 are already in flight is dropped until one finishes. Each transfer finalizes independently (its own EOT trailer + checksum gate), so a slow or interleaved file does not hold up the others.
• Numbered suffix (never overwrite). The output is created with O_CREAT|O_EXCL|O_NOFOLLOW, so an existing name is never overwritten and a symlink at that name is never followed. If <temp>/<fileid> already exists (a previous transfer, or a planted symlink), the receiver writes <fileid>.1, then .2, … up to a bound of 4096; if all are taken the transfer is dropped. The .finished / .corrupt / _checksum.in / _filesize sidecars follow the chosen name.
• Lost-EOT eviction. If a sender dies before sending its EOT trailer, the stalled transfer is evicted after EVICT_TIMEOUT_S seconds of inactivity (default 60), finalized as <name>.corrupt with a reason=eviction line in verify.log, and its resources are freed.
• Write errors fail only the affected transfer, never the daemon. A failed output write — ENOSPC (disk full) or any I/O error — does not crash the receiver. The affected transfer is finalized as <name>.corrupt with a reason=enospc / reason=write-error line in verify.log and its resources freed; the daemon and every other in-flight transfer keep running. verify.log verdict lines carry a reason= field (ok / decode / enospc / write-error / eviction).

Disk planning for parallel transfers. With up to MAX_INFLIGHT (default 8) transfers in flight, peak spool usage is the sum of their output sizes — each is written ~1× in place as it decodes (no 1.3× encoded spool, no separate recovery pass). Size the spool for your expected concurrency; if it fills, only the transfers that hit ENOSPC fail (→ .corrupt reason=enospc), the rest complete, and the senders re-send on the next cycle.

• --progress (both ends, optional, off by default) prints a throttled (~1 line/s) progress line to stderr — useful for long transfers without the old per-group spam. On the receiver it shows decoded/total groups, percent, MB/s and ETA (total/percent/ETA show ? until the EOT trailer arrives, so the receiver knows the group count); it works in --pipe mode too (data on stdout, progress on stderr). On the sender the stdin size is unknown, so it shows only groups sent, bytes sent and MB/s (no percent/ETA). Leading flags may be combined and given in any order, e.g. --pipe --progress. The receiver no longer logs a line per decoded ~10 MB group; use --progress instead.

Logging & configuration

Both binaries emit their [rs] / [sender] operational lines (lifecycle, FILEID rejections, eviction, drops, errors) through a small shared, thread-safe logging module (src/ltlog.[ch]). Each line carries a timestamp and a level; the receiver logs from several threads at once and emission is mutex-serialised, so lines never interleave. The structured per-transfer verify.log journal is unchanged — it keeps its own format on its own path (only the path is now configurable).

Levels

DEBUG < INFO < WARN < ERROR. Default minimum is INFO. Mapping of the existing lines: lifecycle (New transfer, EOT, Created …finished, VERIFIED) → INFO; FILEID rejections, eviction, --pipe second-file / MAX_INFLIGHT drops, decode failures, CORRUPT/DROPPED → WARN; bind/getaddrinfo/alloc/write failures and FATAL → ERROR. Raise/lower the threshold with --log-level <debug|info|warn|error> (or the log_level config key on the receiver).

Destinations

• stderr (default) — unchanged behaviour; under systemd this lands in journald. Format: <ISO-8601+tz> <LEVEL> <tag> message.
• --log <file> — append the same <ts> <LEVEL> <tag> message lines to a file.
• --syslog — syslog(3) with facility LOG_DAEMON (idents lt-diode-receiver / lt-diode-sender). No timestamp is prefixed (syslog/journald adds its own); the level becomes the syslog priority (INFO→LOG_INFO, WARN→LOG_WARNING, ERROR→LOG_ERR).

--progress lines are a separate, throttled feature and still go to stderr as before.

Receiver config file

The receiver daemon can read a simple key = value file (# comments, whitespace trimmed) so it can start with just receiver_stream --config receiver.conf:

receiver_stream --config /etc/lt-diode/receiver.conf

If no --config is given, /etc/lt-diode/receiver.conf is loaded automatically when it exists. Recognised keys: spool (incoming/temp dir — the decoded output, .finished/ .corrupt, sidecars and verify.log all live here), verify_log (path of verify.log; default <spool>/verify.log), log_dest (stderr|file|syslog), log_file (path when log_dest=file), log_level (info by default), port, seed, loss, plus the two runtime tunables max_inflight (parallel-transfer-table depth; default 8, clamped to [1, 256]) and evict_timeout (lost-EOT eviction seconds; default 60, min 1). The compile-time -DMAX_INFLIGHT / -DEVICT_TIMEOUT_S now only set the defaults for these two — the config keys (or --max-inflight / --evict-timeout) override them at runtime (the table is heap-allocated at the chosen depth on startup). See the fully-commented receiver.conf.example.

Precedence

built-in defaults < config file < CLI arguments. A value given on the command line always wins over the same value in the config file, which wins over the compiled-in default. With port, spool, seed and loss all set in the config file the four positional CLI args become optional. The sender is one-shot (CLI only) and has no config file.

Running as a service

The receiver is a long-lived daemon, so it is usually run under an init system. A hardened systemd unit ships in systemd/; SysV-init and OpenRC scripts ship in init/ for non-systemd hosts. All of them just run receiver_stream --config /etc/lt-diode/receiver.conf, so the configuration lives in one place regardless of init system.

The codec is Linux-only (recvmmsg/sendmmsg, OpenMP), so no BSD rc.d or macOS launchd units are provided.

systemd

# build with the unit install rule enabled (the unit dir is distro-specific)
cmake -S . -B build -DLTPRO_INSTALL_SYSTEMD=ON -DLTPRO_SYSTEMD_UNITDIR=/lib/systemd/system
cmake --build build -j && sudo cmake --install build

# drop in the config and start it
sudo install -d /etc/lt-diode
sudo install -m640 receiver.conf.example /etc/lt-diode/receiver.conf
sudo $EDITOR /etc/lt-diode/receiver.conf      # set port/seed/loss; spool + log under /var/lib,/var/log
sudo systemctl daemon-reload
sudo systemctl enable --now lt-diode-receiver

The unit runs unprivileged (DynamicUser), with StateDirectory/LogsDirectory providing /var/lib/lt-diode and /var/log/lt-diode, and is locked down (ProtectSystem=strict, no capabilities, SystemCallFilter=@system-service, MemoryDenyWriteExecute, …). Keep the base port ≥ 1024 so no extra capability is needed.

Tuning the one-way link (qdisc + send buffer)

On a real one-way UDP diode this is the single most important — and most easily missed — tuning. The NIC's default fq_codel / AQM qdisc silently drops the bulk one-way packet blast under load. There is no feedback, so the fountain code cannot recover sender-side drops — those drops are fatal to a transfer, and they show up only in tc -s qdisc show dev <iface> (dropped N); every socket / interface counter stays 0.

Fix: replace the qdisc with a plain pfifo (no AQM). The sender is then paced correctly by a small blocking send buffer (send() blocks at ~one window in flight and self-paces to wire rate). Do not inflate net.core.wmem_max / SO_SNDBUF — a large send buffer removes that backpressure, lets the sender dump everything into the qdisc at once, and makes the AQM drops catastrophic. Backpressure via a small blocking sndbuf is a feature, not a bug.

# one-off (root): plain FIFO, depth in PACKETS (file-size independent)
sudo contrib/diode-tx-tune.sh eth1          # or: LIMIT=2000000 sudo contrib/diode-tx-tune.sh eth1

# persist it (templated by NIC):
sudo install -m755 contrib/diode-tx-tune.sh /usr/local/sbin/
sudo install -m644 systemd/diode-tx-tune@.service /etc/systemd/system/
sudo systemctl enable --now diode-tx-tune@eth1

Verify after a transfer: tc -s qdisc show dev eth1 should report dropped 0.

Repository layout

src/                production streaming codec
  sender_stream.c     zero-temp streaming sender (reads stdin)
  receiver_stream.c   decoupled bounded-RAM receiver (file daemon / --pipe), checksum gate
  ltlog.[ch]          shared thread-safe logging (timestamp + level; stderr/file/syslog)
  rbsoliton.[ch]      robust-soliton degree distribution
  *.h                 protocol / codec headers
  legacy/             upstream-derived, pre-streaming implementations (reference; not built)
CMakeLists.txt        build
systemd/              receiver service unit + diode-tx-tune@.service (qdisc tuning)
init/                 SysV-init and OpenRC service scripts (non-systemd hosts)
contrib/              diode-tx-tune.sh (one-way link qdisc/AQM tuning)
receiver.conf.example annotated receiver daemon config template
CHANGELOG.md          changes vs. the melorian94 upstream
UPSTREAM.md           the original project's README (attribution)
LICENSE               GNU AGPL-3.0

License & attribution

GNU Affero General Public License v3.0 (LICENSE) — same as the upstream. Because it is AGPL, if you run a modified version as part of a network service you must offer its source to the users of that service.

Based on melorian94/robust-soliton-LT-C by Petra — the LT / robust-soliton core, the feedback-free one-way design and the three-port UDP scheme are hers. The modifications layered on top are catalogued in CHANGELOG.md; the upstream README is preserved in UPSTREAM.md.