Enhance mirror_sync with radosgw support - #387
Open
ahresse wants to merge 7 commits into
Open
Conversation
The parallel sync path passed a function defined inside sync_urls to multiprocessing.Pool.map. Local functions cannot be pickled, which broke under Python 3.14 where the default start method on Linux changed from fork to forkserver (tasks are pickled to workers). Hoist _sync_one to module level and pass the closure values through a functools.partial, which is picklable. Signed-off-by: Alexandre Esse <alexandre.esse@canonical.com>
mirror_sync can now populate a radosgw (Ceph RGW) bucket in addition to a local directory tree. When --rgw-host is given, each test suite resource is downloaded to a temp dir and uploaded to the bucket via HTTP PUT, using the same <netloc>/<path> key layout as the local mirror so the bucket URL works directly with `fluster download --mirror`. - add --rgw-host and --bucket options (--bucket required with --rgw-host) - add HEAD-based skip-if-exists check to avoid re-uploading objects - refactor sync_urls to dispatch a per-URL worker for both modes - add bucket-mode tests using a stdlib PUT/HEAD http.server handler Signed-off-by: Alexandre Esse <alexandre.esse@canonical.com>
Update the README Local Mirror section to describe the two mirror_sync backends (local directory and radosgw bucket), add a radosgw usage example with --rgw-host/--bucket, and refresh the mirror_sync.py options help output. Signed-off-by: Alexandre Esse <alexandre.esse@canonical.com>
Signed-off-by: Alexandre Esse <alexandre.esse@canonical.com>
PEP 585 builtin generics (list[str]) are not subscriptable at runtime on Python 3.7, the minimum supported version, causing an ImportError when the module is loaded by tests. Use typing.List instead. Signed-off-by: Alexandre Esse <alexandre.esse@canonical.com>
ahresse
force-pushed
the
http-mirror-upload
branch
from
August 25, 2026 11:09
169fd9e to
aa2840c
Compare
url_to_mirror_path used os.path.join, producing backslashes on Windows that leaked into the radosgw object key (encoded as %5C), breaking the <netloc>/<path> layout that fluster download --mirror expects. Build the key with forward slashes (matching rewrite_url) and convert to OS separators only when writing the local mirror tree. The fake bucket test handler also stored objects at paths containing the host:port ':' character, which is illegal in Windows paths. Map object keys to filesystem-safe paths in the test harness. Signed-off-by: Alexandre Esse <alexandre.esse@canonical.com>
ahresse
force-pushed
the
http-mirror-upload
branch
from
August 25, 2026 11:18
9403403 to
f4fd90b
Compare
Signed-off-by: Alexandre Esse <alexandre.esse@canonical.com>
ahresse
force-pushed
the
http-mirror-upload
branch
from
August 25, 2026 11:27
e9edcda to
9a9282b
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This pull request should be merged after: #386
This pull request adds support for using a radosgw (Ceph RGW) bucket as a mirror backend for test vector resources, in addition to the existing local directory option. It updates both the
scripts/mirror_sync.pyscript and the documentation to describe and implement this new feature, and introduces comprehensive tests for the new bucket upload logic.Major new feature: radosgw (Ceph RGW) bucket mirror support
scripts/mirror_sync.py: Added support for uploading resources to a radosgw bucket over HTTP, including logic for checking object existence, uploading via HTTP PUT, and handling retries. New command-line arguments--rgw-hostand--bucketallow users to specify a bucket as the mirror backend. [1] [2] [3] [4]Documentation updates
README.md: Expanded the documentation to describe how to set up and use a radosgw bucket as a mirror, including example commands and notes on authentication. Updated usage instructions and help text formirror_sync.py. [1] [2] [3]Testing improvements
tests/test_mirror.py: Added new tests for the radosgw bucket upload feature, including a minimal HTTP server to simulate bucket behavior, tests for object upload, skipping existing objects, and URL construction. [1] [2]These changes enable faster and more flexible mirroring of test vectors, especially for distributed or CI environments, by allowing the use of object storage buckets as mirrors.