Skip to content

feat: support virtual-host addressing for S3-compatible snapshot back - #23

Open
davidmyriel wants to merge 6 commits into
kvcache-ai:mainfrom
davidmyriel:myriel/tigris-data
Open

feat: support virtual-host addressing for S3-compatible snapshot back#23
davidmyriel wants to merge 6 commits into
kvcache-ai:mainfrom
davidmyriel:myriel/tigris-data

Conversation

@davidmyriel

Copy link
Copy Markdown

What problem does this solve?

I tried to point the OSS snapshot backend at an S3-compatible provider that requires virtual-host bucket addressing and found there was no way to make it work.

The backend picks the addressing style automatically in detect_addressing_style: Alibaba OSS endpoints and endpoints that already contain the bucket name get virtual-host style (https://bucket.example.com/key), and everything else falls back to path style (https://example.com/bucket/key). That fallback is the problem — a growing set of providers only reliably support virtual-host addressing (AWS S3 buckets created after the path-style deprecation, Cloudflare R2, Tigris), and for those the auto-detection guesses wrong with no way to override it. The result is confusing request failures at runtime rather than anything that points at addressing style.

What this PR does

Adds an optional addressing_style field to [backend.oss]:

[backend.oss]
endpoint = "https://t3.storage.dev"
bucket = "agentenv-snapshots"
region = "auto"
addressing_style = "virtual"   # or "path"

The behavior is strictly opt-in:

  • When set, the explicit value wins and is passed straight through to the object-store client.
  • When unset, the existing detect_addressing_style auto-detection runs exactly as before — zero behavior change for current Alibaba OSS / MinIO setups.

The plumbing is small: a new OssAddressingStyle config enum, a mapping in NormalizedOssConfig, and an override-or-detect resolution in OssClient::new.

Testing

  • Unit tests covering: the detection fallback for generic endpoints, the existing Aliyun/bucket-in-endpoint detection, the override taking precedence, and the config mapping.
  • Verified end-to-end against a real Tigris bucket with the included tigris_smoke example, which drives the same build_object_store_operator path the backend uses: write → read-back → verify → delete round-trip succeeds with addressing_style = "virtual".
  • Also confirmed at the wire level (via a local Host-header echo endpoint) that the two styles genuinely produce different requests — Host: bucket.endpoint vs /bucket/key on the path — so the override reaches the wire rather than being silently dropped.
  • cargo fmt, cargo clippy -- -D warnings, and cargo test -p object-store-operator are clean.

Docs

  • docs/src/configuration/reference.md: new row in the [backend.oss] table plus a short worked example.
  • docs/src/getting-started/on-demand-loading.md: one-line pointer after the OSS example config, since anyone following that guide with a virtual-host-only provider would otherwise hit an opaque failure.
  • Commented-out example in config/default.toml.

Happy to drop the smoke example if you'd rather keep examples/ lean

@github-actions

github-actions Bot commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

OpenCodeReview: No comments generated. Looks good to me.

Comment thread crates/object-store-operator/examples/tigris_smoke.rs Outdated
Comment thread crates/object-store-operator/examples/tigris_smoke.rs Outdated
Comment thread src/cfg.rs
Comment thread src/snapshot/repository/backends/oss/client.rs Outdated
@yingdi-shan
yingdi-shan requested a review from guozy18 July 28, 2026 01:51

@guozy18 guozy18 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for adding this. Being able to override the addressing style is useful for S3-compatible providers that cannot use the current default.

I'm requesting changes because the setting currently only reaches the repository client, not the OverlayBD path used to read remote snapshot layers. That means uploads can succeed while restoring a sandbox still fails. There is also a missing field in an existing integration test, plus a couple of cleanup issues in the smoke example.

One more thing: could you also update the commit message to follow the Conventional Commits format? The current message, "add tigris data", does not describe the change clearly. Something like feat(oss): support configurable S3 addressing style would be more appropriate. See https://www.conventionalcommits.org/en/v1.0.0/.

Comment thread src/snapshot/repository/backends/oss/mod.rs
Comment thread src/cfg.rs
Comment thread src/snapshot/repository/backends/oss/client.rs Outdated
Comment thread crates/object-store-operator/examples/tigris_smoke.rs Outdated
Comment thread crates/object-store-operator/examples/tigris_smoke.rs Outdated
davidmyriel and others added 2 commits July 29, 2026 11:23
Address review feedback:
- propagate backend.oss.addressing_style into the generated overlaybd
  runtime config (ossConfig.defaultAddressingStyle) so remote managed
  snapshot layer reads use the same style as the repository client
- always run detect_addressing_style for endpoint validation, then apply
  the explicit override on top (repository client and overlaybd URL parse)
- validate defaultAddressingStyle at overlaybd config load
- add addressing_style to the OssBackendConfig literal in
  tests/snapshot_oss_e2e_test.rs
- smoke example: collision-resistant per-run object key, and always
  attempt cleanup once the write succeeded, surfacing the primary error
- cover the above with unit tests plus an ignored MinIO remote-layer
  read test using an explicit addressing style

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Comment thread crates/object-store-operator/examples/tigris_smoke.rs Outdated
Comment thread crates/object-store-operator/examples/tigris_smoke.rs Outdated
Comment thread crates/object-store-operator/examples/tigris_smoke.rs Outdated
Comment thread storage/overlaybd/tests/oss_backend_minio.rs Outdated
Address second-round review feedback:
- smoke example: propagate missing env vars as errors instead of
  panicking, use a random UUID object key so concurrent runs on any
  host cannot collide, and attempt cleanup even after a failed write
  since a timeout can leave the object committed remotely
- replace the MinIO explicit-style test (which exercised a combination
  auto-detection would have picked anyway) with a wire-level test:
  bucket '127' on a 127.0.0.1 endpoint auto-detects as virtual-host, so
  an explicit 'path' override observably flips the recorded request to
  path style; the test fails if the override is dropped anywhere
- also assert invalid defaultAddressingStyle values are rejected at
  backend construction

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Comment thread crates/object-store-operator/examples/tigris_smoke.rs Outdated
Comment thread crates/object-store-operator/examples/tigris_smoke.rs Outdated
Comment thread storage/overlaybd/tests/oss_addressing_style.rs Outdated
Comment thread storage/overlaybd/tests/oss_addressing_style.rs Outdated
The smoke example was a manual verification aid, not part of the
feature; removing it (and its uuid dev-dependency) keeps the PR scoped
to the addressing-style change. The wire-level propagation test stays
and is hardened per review: the recorder now reads until the header
terminator so partial TCP reads cannot flake the assertion, and a
single timeout bounds the whole read + capture interaction.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@davidmyriel
davidmyriel requested a review from guozy18 July 29, 2026 16:12
@davidmyriel

Copy link
Copy Markdown
Author

@guozy18 all the comments are addressed and verified via test. Should be good to go.

I dropped the smoke test from the PR as it was attracting Github bots comments.

Congrats on the project success. Would like to do some promotions and comarketing + more integration work.

Comment thread storage/overlaybd/tests/oss_addressing_style.rs
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Comment thread config/default.toml
Comment on lines +153 to +156
# # path style). Set "virtual" for providers that require virtual-host
# # addressing, e.g. AWS S3 new buckets, Cloudflare R2, or Tigris
# # (endpoint = "https://t3.storage.dev", region = "auto").
# addressing_style = "virtual"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[documentation · low]
The new option also accepts "path", but the default configuration only documents the virtual override. An explicit path override is important when auto-detection selects virtual style (for example, an Alibaba/custom bucket-in-host endpoint that nevertheless expects path style). Please list both accepted values and describe "path" as the force-path override; also consider avoiding “require” for the provider examples unless that requirement is guaranteed, since some S3-compatible providers support both styles.

Suggestion:

Suggested change
# # path style). Set "virtual" for providers that require virtual-host
# # addressing, e.g. AWS S3 new buckets, Cloudflare R2, or Tigris
# # (endpoint = "https://t3.storage.dev", region = "auto").
# addressing_style = "virtual"
# # path style). Set "virtual" to force virtual-host addressing, or "path"
# # to force path-style addressing. For example, virtual-host style can be used
# # with AWS S3, Cloudflare R2, or Tigris
# # (endpoint = "https://t3.storage.dev", region = "auto").
# addressing_style = "virtual"

Comment on lines +69 to +70
.open_with_size_hint("s3://127/layers/explicit-style-object", None)
.expect("open remote layer file");

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[test · low]
Passing None makes read_at call size() first, so the recorder's only request is a metadata HEAD; the 404 then prevents the advertised range GET from being issued. This still checks the operator host/path, but it does not provide the stated wire-level proof for remote layer reads. Supply a size hint so read_at sends the GET directly (and consider asserting the request method).

Suggestion:

Suggested change
.open_with_size_hint("s3://127/layers/explicit-style-object", None)
.expect("open remote layer file");
.open_with_size_hint("s3://127/layers/explicit-style-object", Some(4))
.expect("open remote layer file");

Comment thread storage/overlaybd/tests/oss_addressing_style.rs Outdated
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

@guozy18 guozy18 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the kind words, and thanks for addressing the earlier feedback. The main implementation direction now looks right, while the existing auto-detection behavior remains the default.

I still have a few requests before this is ready to merge:

  1. Rebase onto the latest main and fix the resulting build failure

  2. Address the two remaining automated review comments

    Both look valid to me:

    • Document both supported overrides, "virtual" and "path", and avoid overstating which providers strictly require either style.
    • Change the remote-read test from None to Some(4) so it exercises the range GET path rather than stopping after a metadata HEAD request. It would also be useful to assert that the captured request method is GET.
  3. Do a simplification pass over the implementation and tests

    The PR currently adds around 375 lines, with a substantial amount of overlapping test coverage. I would keep the tests that protect distinct contracts: configuration parsing, repository-client override behavior, propagation into the generated OverlayBD config, endpoint validation, and one wire-level remote-read test.

    Tests that only cover unchanged auto-detection behavior, or repeat the same override behavior at adjacent layers, can likely be consolidated. For example, the newly added repository-client auto-detection tests overlap with existing OverlayBD tests, and some of the parsing, normalization, and wire-level assertions cover the same branch more than once.

Comment on lines 31 to 37
credentials: CachedCredentialSource,
default_region: String,
default_endpoint: String,
addressing_override: Option<AddressingStyle>,
timeout: Duration,
retry_count: u32,
cached_operators: RwLock<HashMap<OperatorCacheKey, OperatorWithCredential>>,

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR changes OssClient::new to take six arguments. However, the snapshot image export service recently added on main still calls it with five arguments in image_export/service.rs.

After rebasing, please pass the configured override there as well:

config.addressing_style(),

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants