Skip to content

Git object-store A3 conformance probe fails on GCS S3-interop (if_match_race) — consider a native GCS backend #2470

Description

@vsima

Environment

  • Relay: ghcr.io/block/buzz:sha-2d26db6 (2d26db6d8a7381ff0dc9b50194fd28d42957de74); reproduced identically on sha-acfbb1b (2026-07-22 main)
  • Object storage: Google Cloud Storage via the S3-interop XML API (BUZZ_S3_ENDPOINT=https://storage.googleapis.com, HMAC service-account credentials)
  • Deployment: GKE, chart buzz 0.1.6, external Postgres/Redis

What happens

Relay startup is fatal at the A3 git conformance gate:

INFO  running git object-store conformance probe (A3 gate)
Error: git conformance probe failed: conformance probe failed in phase 'if_match_race' (round 0, key probe/pointer-0d592977-…)

Media paths on the same bucket/credentials work (Media storage connected;
put/get verified end-to-end through the app).

Why

The probe is right to fail: GCS's S3-interoperability layer does not honor
S3-style conditional writes (If-Match/If-None-Match preconditions on PUT),
so the linearizable pointer CAS that run_conformance_probe verifies (and
that the manifest-pointer protocol needs) cannot be satisfied. This makes
git-on-object-storage unusable on GCS for any deployer using the documented
S3 configuration — the probe converts that into a hard startup failure.

What we did (and why it's not a fix)

Booted the pilot with BUZZ_GIT_CONFORMANCE_PROBE=false and quarantined git
features operationally. That's an availability workaround, not a solution —
the probe's verdict about pointer CAS still stands.

Proposal

GCS does support real CAS — via its native preconditions
(x-goog-if-generation-match: 0 for create-if-absent, generation-match for
compare-and-swap). Options in rough order of preference:

  1. Native GCS storage backend for the git object store (JSON/XML API with
    generation preconditions) behind the existing storage trait.
  2. Conditional-write shim for GCS endpoints: detect
    storage.googleapis.com and translate the pointer-CAS operations to
    x-goog-* preconditions while leaving plain put/get on the S3 path.
  3. At minimum, document that git-on-object-storage requires a backend with
    S3 conditional-write semantics (AWS S3 ≥ 2024 If-Match support, MinIO, R2?)
    and name GCS S3-interop as unsupported — so deployers hit a doc instead of
    a fatal probe.

We (Apiary — hosted Buzz on GCP) are happy to test any of these against real
GCS, and would rather contribute the backend than run a MinIO shim in front
of GCS.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions