feat: support virtual-host addressing for S3-compatible snapshot back - #23
feat: support virtual-host addressing for S3-compatible snapshot back#23davidmyriel wants to merge 6 commits into
Conversation
|
✅ OpenCodeReview: No comments generated. Looks good to me. |
guozy18
left a comment
There was a problem hiding this comment.
Thanks for adding this. Being able to override the addressing style is useful for S3-compatible providers that cannot use the current default.
I'm requesting changes because the setting currently only reaches the repository client, not the OverlayBD path used to read remote snapshot layers. That means uploads can succeed while restoring a sandbox still fails. There is also a missing field in an existing integration test, plus a couple of cleanup issues in the smoke example.
One more thing: could you also update the commit message to follow the Conventional Commits format? The current message, "add tigris data", does not describe the change clearly. Something like feat(oss): support configurable S3 addressing style would be more appropriate. See https://www.conventionalcommits.org/en/v1.0.0/.
Address review feedback: - propagate backend.oss.addressing_style into the generated overlaybd runtime config (ossConfig.defaultAddressingStyle) so remote managed snapshot layer reads use the same style as the repository client - always run detect_addressing_style for endpoint validation, then apply the explicit override on top (repository client and overlaybd URL parse) - validate defaultAddressingStyle at overlaybd config load - add addressing_style to the OssBackendConfig literal in tests/snapshot_oss_e2e_test.rs - smoke example: collision-resistant per-run object key, and always attempt cleanup once the write succeeded, surfacing the primary error - cover the above with unit tests plus an ignored MinIO remote-layer read test using an explicit addressing style Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
98a2eab to
b54d04a
Compare
Address second-round review feedback: - smoke example: propagate missing env vars as errors instead of panicking, use a random UUID object key so concurrent runs on any host cannot collide, and attempt cleanup even after a failed write since a timeout can leave the object committed remotely - replace the MinIO explicit-style test (which exercised a combination auto-detection would have picked anyway) with a wire-level test: bucket '127' on a 127.0.0.1 endpoint auto-detects as virtual-host, so an explicit 'path' override observably flips the recorded request to path style; the test fails if the override is dropped anywhere - also assert invalid defaultAddressingStyle values are rejected at backend construction Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The smoke example was a manual verification aid, not part of the feature; removing it (and its uuid dev-dependency) keeps the PR scoped to the addressing-style change. The wire-level propagation test stays and is hardened per review: the recorder now reads until the header terminator so partial TCP reads cannot flake the assertion, and a single timeout bounds the whole read + capture interaction. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
@guozy18 all the comments are addressed and verified via test. Should be good to go. I dropped the smoke test from the PR as it was attracting Github bots comments. Congrats on the project success. Would like to do some promotions and comarketing + more integration work. |
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
| # # path style). Set "virtual" for providers that require virtual-host | ||
| # # addressing, e.g. AWS S3 new buckets, Cloudflare R2, or Tigris | ||
| # # (endpoint = "https://t3.storage.dev", region = "auto"). | ||
| # addressing_style = "virtual" |
There was a problem hiding this comment.
[documentation · low]
The new option also accepts "path", but the default configuration only documents the virtual override. An explicit path override is important when auto-detection selects virtual style (for example, an Alibaba/custom bucket-in-host endpoint that nevertheless expects path style). Please list both accepted values and describe "path" as the force-path override; also consider avoiding “require” for the provider examples unless that requirement is guaranteed, since some S3-compatible providers support both styles.
Suggestion:
| # # path style). Set "virtual" for providers that require virtual-host | |
| # # addressing, e.g. AWS S3 new buckets, Cloudflare R2, or Tigris | |
| # # (endpoint = "https://t3.storage.dev", region = "auto"). | |
| # addressing_style = "virtual" | |
| # # path style). Set "virtual" to force virtual-host addressing, or "path" | |
| # # to force path-style addressing. For example, virtual-host style can be used | |
| # # with AWS S3, Cloudflare R2, or Tigris | |
| # # (endpoint = "https://t3.storage.dev", region = "auto"). | |
| # addressing_style = "virtual" |
| .open_with_size_hint("s3://127/layers/explicit-style-object", None) | ||
| .expect("open remote layer file"); |
There was a problem hiding this comment.
[test · low]
Passing None makes read_at call size() first, so the recorder's only request is a metadata HEAD; the 404 then prevents the advertised range GET from being issued. This still checks the operator host/path, but it does not provide the stated wire-level proof for remote layer reads. Supply a size hint so read_at sends the GET directly (and consider asserting the request method).
Suggestion:
| .open_with_size_hint("s3://127/layers/explicit-style-object", None) | |
| .expect("open remote layer file"); | |
| .open_with_size_hint("s3://127/layers/explicit-style-object", Some(4)) | |
| .expect("open remote layer file"); |
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
guozy18
left a comment
There was a problem hiding this comment.
Thanks for the kind words, and thanks for addressing the earlier feedback. The main implementation direction now looks right, while the existing auto-detection behavior remains the default.
I still have a few requests before this is ready to merge:
-
Rebase onto the latest main and fix the resulting build failure
-
Address the two remaining automated review comments
Both look valid to me:
- Document both supported overrides,
"virtual"and"path", and avoid overstating which providers strictly require either style. - Change the remote-read test from
NonetoSome(4)so it exercises the range GET path rather than stopping after a metadata HEAD request. It would also be useful to assert that the captured request method is GET.
- Document both supported overrides,
-
Do a simplification pass over the implementation and tests
The PR currently adds around 375 lines, with a substantial amount of overlapping test coverage. I would keep the tests that protect distinct contracts: configuration parsing, repository-client override behavior, propagation into the generated OverlayBD config, endpoint validation, and one wire-level remote-read test.
Tests that only cover unchanged auto-detection behavior, or repeat the same override behavior at adjacent layers, can likely be consolidated. For example, the newly added repository-client auto-detection tests overlap with existing OverlayBD tests, and some of the parsing, normalization, and wire-level assertions cover the same branch more than once.
| credentials: CachedCredentialSource, | ||
| default_region: String, | ||
| default_endpoint: String, | ||
| addressing_override: Option<AddressingStyle>, | ||
| timeout: Duration, | ||
| retry_count: u32, | ||
| cached_operators: RwLock<HashMap<OperatorCacheKey, OperatorWithCredential>>, |
There was a problem hiding this comment.
This PR changes OssClient::new to take six arguments. However, the snapshot image export service recently added on main still calls it with five arguments in image_export/service.rs.
After rebasing, please pass the configured override there as well:
config.addressing_style(),
What problem does this solve?
I tried to point the OSS snapshot backend at an S3-compatible provider that requires virtual-host bucket addressing and found there was no way to make it work.
The backend picks the addressing style automatically in
detect_addressing_style: Alibaba OSS endpoints and endpoints that already contain the bucket name get virtual-host style (https://bucket.example.com/key), and everything else falls back to path style (https://example.com/bucket/key). That fallback is the problem — a growing set of providers only reliably support virtual-host addressing (AWS S3 buckets created after the path-style deprecation, Cloudflare R2, Tigris), and for those the auto-detection guesses wrong with no way to override it. The result is confusing request failures at runtime rather than anything that points at addressing style.What this PR does
Adds an optional
addressing_stylefield to[backend.oss]:The behavior is strictly opt-in:
detect_addressing_styleauto-detection runs exactly as before — zero behavior change for current Alibaba OSS / MinIO setups.The plumbing is small: a new
OssAddressingStyleconfig enum, a mapping inNormalizedOssConfig, and an override-or-detect resolution inOssClient::new.Testing
tigris_smokeexample, which drives the samebuild_object_store_operatorpath the backend uses: write → read-back → verify → delete round-trip succeeds withaddressing_style = "virtual".Host: bucket.endpointvs/bucket/keyon the path — so the override reaches the wire rather than being silently dropped.cargo fmt,cargo clippy -- -D warnings, andcargo test -p object-store-operatorare clean.Docs
docs/src/configuration/reference.md: new row in the[backend.oss]table plus a short worked example.docs/src/getting-started/on-demand-loading.md: one-line pointer after the OSS example config, since anyone following that guide with a virtual-host-only provider would otherwise hit an opaque failure.config/default.toml.Happy to drop the smoke example if you'd rather keep
examples/lean