SSH transfer optimizations: source-side compression and stale staging purge (v3.10.0) - #3
Merged
Merged
Conversation
…v3.10.0) - --compress-where source|target (default: target, old behavior): with zstd/xz/bzip2, compress the dump on the SSH source host before the transfer, so slow uplinks move the compressed artifact instead of the raw dump. Falls back to target-side compression with a warning when the tool is missing on the source or compression fails. gzip is unaffected (pg_dump -Z already compresses in-format at the source). - --from-stale <time> (default: 72h, 0 disables): with --from-keep 0, purge this job's leftover dumps older than <time> from the source staging dir before dumping. Failed or interrupted runs never reach the post-transfer cleanup, so their dumps piled up in /tmp forever. - Widen the --from-keep retention and fetch latest-dump globs from _*.dump to _*.dump* so compressed staging artifacts are matched. - Both options are validated (enum / strict time format), supported in config files and jobs.yaml (save_job), and documented in --help and all READMEs. Verified end-to-end against a postgres:16+sshd container: source-side compression (pg_restore -l integrity check on the transferred artifact), missing-tool fallback, stale purge scoping (only this job's old files), from-keep retention with compressed names, and unchanged default-path behavior.
The batch runner translates YAML defaults/job keys to child CLI args via an explicit key list in yaml.sh, which the new options were missing from — a compress_where/from_stale set in a jobs YAML was silently ignored.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two features for SSH dump jobs on slow links, plus a fix they surfaced:
--compress-where source|target(defaulttarget, old behavior): with zstd/xz/bzip2, compress the dump on the source host before the transfer, so slow uplinks move the compressed artifact instead of the raw dump (20x+ smaller in practice). Falls back to target-side compression with a warning when the tool is missing on the source or compression fails — never blocks the backup. gzip is unaffected (pg_dump -Z already compresses in-format at the source).--from-stale <time>(default72h,0disables): with--from-keep 0, purge this job's leftover dumps older than<time>from the source staging dir before dumping. Failed or interrupted runs never reach the post-transfer cleanup, so their dumps piled up in the source /tmp forever.--from-keepretention and fetch globs from_*.dumpto_*.dump*so compressed staging artifacts are matched.Both options are validated (enum / strict time format — config and YAML values pass through the same validation), supported in config files and jobs YAML, and documented in
--helpand all four READMEs.Testing
Verified end-to-end against a postgres:16 + sshd container:
.dump.zst,pg_restore -lintegrity check on the decompressed artifact, source staging cleaned--from-keep 1retention with compressed staging names--compress-where) byte-for-byte old behaviorcompress_where/from_stalevalues in a jobs YAML are rejected by the child validation (proves the keys flow end-to-end)./build.sh --checkgreen,bash -nclean