Skip to content

Use a trait for remote ByteStore network implementation - #19050

Merged
huonw merged 6 commits into
pantsbuild:mainfrom
huonw:feature/byte-store-trait-2
May 23, 2023
Merged

Use a trait for remote ByteStore network implementation#19050
huonw merged 6 commits into
pantsbuild:mainfrom
huonw:feature/byte-store-trait-2

Conversation

@huonw

@huonw huonw commented May 19, 2023

Copy link
Copy Markdown
Contributor

This separates the store::remote::ByteStore coordination code from the gRPC REAPI implementation code, by:

  • adding a store::remote::ByteStoreProvider trait
  • moving the REAPI implementation into store::remote::reapi and implementing that trait

This is, in theory, just a lift-and-shift, with no functionality change, other than maybe changing some logging and when exactly a work-unit is created.

This is partial preparation work for supporting more remote stores like GHA cache or S3, for #11149, and is specifically broken out of #17840. Additional work required to actually solve #11149:

  • generalising the action cache too
  • implementing other byte store providers
  • dynamically choosing the right provider for store::remote::ByteStore

The commits are individually reviewable:

  1. preparatory clean-up
  2. define the trait
  3. move the REAPI code and implement the trait, close to naively as possible:
  4. minor clean-up
  5. more major clean-up: inline the method calls into the trait definition, so that they're not just fn method(&self, ...) { self.real_method(...) }:
    • as with 3, this is just a move, without changing the details of the code...
    • ...except for some minor changes like converting ? into .map(|e| e.to_string())? on the get_capabilities call in store_bytes

@huonw huonw added the category:internal CI, fixes for not-yet-released features, etc. label May 19, 2023
@huonw
huonw force-pushed the feature/byte-store-trait-2 branch from 91b6c30 to 1800f55 Compare May 19, 2023 07:34
@huonw
huonw marked this pull request as ready for review May 19, 2023 07:37
@huonw
huonw force-pushed the feature/byte-store-trait-2 branch from 1800f55 to 9bdf435 Compare May 19, 2023 11:11
Comment thread src/rust/engine/fs/store/src/remote/reapi.rs Outdated
Comment thread src/rust/engine/fs/store/src/remote/reapi.rs Outdated
@huonw
huonw merged commit cc8be36 into pantsbuild:main May 23, 2023
@huonw
huonw deleted the feature/byte-store-trait-2 branch May 23, 2023 00:55
illicitonion added a commit to illicitonion/pants that referenced this pull request May 26, 2023
Internal changes:

* Fix TypedDict construction ([pantsbuild#19166](pantsbuild#19166))

* Have package_dists.InvalidEntryPoint subclass InvalidFieldException ([pantsbuild#19149](pantsbuild#19149))

* Check for `CI=true`, not `CI=1` for cherry-picking ([pantsbuild#19141](pantsbuild#19141))

* Set up remote caching in CI  ([pantsbuild#19144](pantsbuild#19144))

* refactor(internal): adjust pants_venv venv_dir calculation ([pantsbuild#19140](pantsbuild#19140))

* Prepare `2.16.0rc4`. ([pantsbuild#19146](pantsbuild#19146))

* Warn about `files()` in PEXes in one place ([pantsbuild#19027](pantsbuild#19027))

* Drop our explicit dep on `syn` ([pantsbuild#19134](pantsbuild#19134))

* Fix auto-cherry-picker's merge-conflicts ([pantsbuild#19125](pantsbuild#19125))

* Use explicit depths on fetches when running cherry-picks in CI ([pantsbuild#19127](pantsbuild#19127))

* Make determine_python() pin to 3.9 ([pantsbuild#19116](pantsbuild#19116))

* Upgrade to `actions/checkout@v3` ([pantsbuild#19112](pantsbuild#19112))

* Run the cherry pick script with `-x` for debugging ([pantsbuild#19095](pantsbuild#19095))

* Add more linux x86 test CI shards ([pantsbuild#19099](pantsbuild#19099))

* Use a trait for remote ByteStore network implementation ([pantsbuild#19050](pantsbuild#19050))

* Increase the logging of release version file publishing. ([pantsbuild#19092](pantsbuild#19092))

* Remove the Toolchain plugin ([pantsbuild#18978](pantsbuild#18978))

* Fix bad merge of pantsbuild#18829. ([pantsbuild#19080](pantsbuild#19080))

* Revert "Implement layout="zip" for Lambda/GCF, skipping lambdex" ([pantsbuild#19071](pantsbuild#19071))

* Remove unnecessary interpreter constraint config. ([pantsbuild#19065](pantsbuild#19065))

* Use hand-written change detection for PR/push job trimming ([pantsbuild#19045](pantsbuild#19045))

* Do not build wheels on push, since that is now accomplished by the release job. ([pantsbuild#19063](pantsbuild#19063))

* Additional release job publishing fixes ([pantsbuild#19058](pantsbuild#19058))

* Change CI trigger to includelist `main` and `2.*.x` ([pantsbuild#19059](pantsbuild#19059))

* Release doc and script fixes ([pantsbuild#19043](pantsbuild#19043))

* Update cherry-picker to use `pull_request_target` ([pantsbuild#19048](pantsbuild#19048))

* Prepare `2.16.0rc3`. ([pantsbuild#19044](pantsbuild#19044))

* Prepare `2.15.1`. ([pantsbuild#19042](pantsbuild#19042))

* Fixup welcome-newcomers.yaml ([pantsbuild#19039](pantsbuild#19039))
@illicitonion illicitonion mentioned this pull request May 26, 2023
illicitonion added a commit to illicitonion/pants that referenced this pull request May 26, 2023
Internal changes:

* Fix TypedDict construction ([pantsbuild#19166](pantsbuild#19166))

* Have package_dists.InvalidEntryPoint subclass InvalidFieldException ([pantsbuild#19149](pantsbuild#19149))

* Check for `CI=true`, not `CI=1` for cherry-picking ([pantsbuild#19141](pantsbuild#19141))

* Set up remote caching in CI  ([pantsbuild#19144](pantsbuild#19144))

* refactor(internal): adjust pants_venv venv_dir calculation ([pantsbuild#19140](pantsbuild#19140))

* Prepare `2.16.0rc4`. ([pantsbuild#19146](pantsbuild#19146))

* Warn about `files()` in PEXes in one place ([pantsbuild#19027](pantsbuild#19027))

* Drop our explicit dep on `syn` ([pantsbuild#19134](pantsbuild#19134))

* Fix auto-cherry-picker's merge-conflicts ([pantsbuild#19125](pantsbuild#19125))

* Use explicit depths on fetches when running cherry-picks in CI ([pantsbuild#19127](pantsbuild#19127))

* Make determine_python() pin to 3.9 ([pantsbuild#19116](pantsbuild#19116))

* Upgrade to `actions/checkout@v3` ([pantsbuild#19112](pantsbuild#19112))

* Run the cherry pick script with `-x` for debugging ([pantsbuild#19095](pantsbuild#19095))

* Add more linux x86 test CI shards ([pantsbuild#19099](pantsbuild#19099))

* Use a trait for remote ByteStore network implementation ([pantsbuild#19050](pantsbuild#19050))

* Increase the logging of release version file publishing. ([pantsbuild#19092](pantsbuild#19092))

* Remove the Toolchain plugin ([pantsbuild#18978](pantsbuild#18978))

* Fix bad merge of pantsbuild#18829. ([pantsbuild#19080](pantsbuild#19080))

* Revert "Implement layout="zip" for Lambda/GCF, skipping lambdex" ([pantsbuild#19071](pantsbuild#19071))

* Remove unnecessary interpreter constraint config. ([pantsbuild#19065](pantsbuild#19065))

* Use hand-written change detection for PR/push job trimming ([pantsbuild#19045](pantsbuild#19045))

* Do not build wheels on push, since that is now accomplished by the release job. ([pantsbuild#19063](pantsbuild#19063))

* Additional release job publishing fixes ([pantsbuild#19058](pantsbuild#19058))

* Change CI trigger to includelist `main` and `2.*.x` ([pantsbuild#19059](pantsbuild#19059))

* Release doc and script fixes ([pantsbuild#19043](pantsbuild#19043))

* Update cherry-picker to use `pull_request_target` ([pantsbuild#19048](pantsbuild#19048))

* Prepare `2.16.0rc3`. ([pantsbuild#19044](pantsbuild#19044))

* Prepare `2.15.1`. ([pantsbuild#19042](pantsbuild#19042))

* Fixup welcome-newcomers.yaml ([pantsbuild#19039](pantsbuild#19039))
illicitonion added a commit to illicitonion/pants that referenced this pull request May 26, 2023
Internal changes:

* Fix TypedDict construction ([pantsbuild#19166](pantsbuild#19166))

* Have package_dists.InvalidEntryPoint subclass InvalidFieldException ([pantsbuild#19149](pantsbuild#19149))

* Check for `CI=true`, not `CI=1` for cherry-picking ([pantsbuild#19141](pantsbuild#19141))

* Set up remote caching in CI  ([pantsbuild#19144](pantsbuild#19144))

* refactor(internal): adjust pants_venv venv_dir calculation ([pantsbuild#19140](pantsbuild#19140))

* Prepare `2.16.0rc4`. ([pantsbuild#19146](pantsbuild#19146))

* Warn about `files()` in PEXes in one place ([pantsbuild#19027](pantsbuild#19027))

* Drop our explicit dep on `syn` ([pantsbuild#19134](pantsbuild#19134))

* Fix auto-cherry-picker's merge-conflicts ([pantsbuild#19125](pantsbuild#19125))

* Use explicit depths on fetches when running cherry-picks in CI ([pantsbuild#19127](pantsbuild#19127))

* Make determine_python() pin to 3.9 ([pantsbuild#19116](pantsbuild#19116))

* Upgrade to `actions/checkout@v3` ([pantsbuild#19112](pantsbuild#19112))

* Run the cherry pick script with `-x` for debugging ([pantsbuild#19095](pantsbuild#19095))

* Add more linux x86 test CI shards ([pantsbuild#19099](pantsbuild#19099))

* Use a trait for remote ByteStore network implementation ([pantsbuild#19050](pantsbuild#19050))

* Increase the logging of release version file publishing. ([pantsbuild#19092](pantsbuild#19092))

* Remove the Toolchain plugin ([pantsbuild#18978](pantsbuild#18978))

* Fix bad merge of pantsbuild#18829. ([pantsbuild#19080](pantsbuild#19080))

* Revert "Implement layout="zip" for Lambda/GCF, skipping lambdex" ([pantsbuild#19071](pantsbuild#19071))

* Remove unnecessary interpreter constraint config. ([pantsbuild#19065](pantsbuild#19065))

* Use hand-written change detection for PR/push job trimming ([pantsbuild#19045](pantsbuild#19045))

* Do not build wheels on push, since that is now accomplished by the release job. ([pantsbuild#19063](pantsbuild#19063))

* Additional release job publishing fixes ([pantsbuild#19058](pantsbuild#19058))

* Change CI trigger to includelist `main` and `2.*.x` ([pantsbuild#19059](pantsbuild#19059))

* Release doc and script fixes ([pantsbuild#19043](pantsbuild#19043))

* Update cherry-picker to use `pull_request_target` ([pantsbuild#19048](pantsbuild#19048))

* Prepare `2.16.0rc3`. ([pantsbuild#19044](pantsbuild#19044))

* Prepare `2.15.1`. ([pantsbuild#19042](pantsbuild#19042))

* Fixup welcome-newcomers.yaml ([pantsbuild#19039](pantsbuild#19039))
illicitonion added a commit to illicitonion/pants that referenced this pull request May 26, 2023
Internal changes:

* Fix TypedDict construction ([pantsbuild#19166](pantsbuild#19166))

* Have package_dists.InvalidEntryPoint subclass InvalidFieldException ([pantsbuild#19149](pantsbuild#19149))

* Check for `CI=true`, not `CI=1` for cherry-picking ([pantsbuild#19141](pantsbuild#19141))

* Set up remote caching in CI  ([pantsbuild#19144](pantsbuild#19144))

* refactor(internal): adjust pants_venv venv_dir calculation ([pantsbuild#19140](pantsbuild#19140))

* Prepare `2.16.0rc4`. ([pantsbuild#19146](pantsbuild#19146))

* Warn about `files()` in PEXes in one place ([pantsbuild#19027](pantsbuild#19027))

* Drop our explicit dep on `syn` ([pantsbuild#19134](pantsbuild#19134))

* Fix auto-cherry-picker's merge-conflicts ([pantsbuild#19125](pantsbuild#19125))

* Use explicit depths on fetches when running cherry-picks in CI ([pantsbuild#19127](pantsbuild#19127))

* Make determine_python() pin to 3.9 ([pantsbuild#19116](pantsbuild#19116))

* Upgrade to `actions/checkout@v3` ([pantsbuild#19112](pantsbuild#19112))

* Run the cherry pick script with `-x` for debugging ([pantsbuild#19095](pantsbuild#19095))

* Add more linux x86 test CI shards ([pantsbuild#19099](pantsbuild#19099))

* Use a trait for remote ByteStore network implementation ([pantsbuild#19050](pantsbuild#19050))

* Increase the logging of release version file publishing. ([pantsbuild#19092](pantsbuild#19092))

* Remove the Toolchain plugin ([pantsbuild#18978](pantsbuild#18978))

* Fix bad merge of pantsbuild#18829. ([pantsbuild#19080](pantsbuild#19080))

* Revert "Implement layout="zip" for Lambda/GCF, skipping lambdex" ([pantsbuild#19071](pantsbuild#19071))

* Remove unnecessary interpreter constraint config. ([pantsbuild#19065](pantsbuild#19065))

* Use hand-written change detection for PR/push job trimming ([pantsbuild#19045](pantsbuild#19045))

* Do not build wheels on push, since that is now accomplished by the release job. ([pantsbuild#19063](pantsbuild#19063))

* Additional release job publishing fixes ([pantsbuild#19058](pantsbuild#19058))

* Change CI trigger to includelist `main` and `2.*.x` ([pantsbuild#19059](pantsbuild#19059))

* Release doc and script fixes ([pantsbuild#19043](pantsbuild#19043))

* Update cherry-picker to use `pull_request_target` ([pantsbuild#19048](pantsbuild#19048))

* Prepare `2.16.0rc3`. ([pantsbuild#19044](pantsbuild#19044))

* Prepare `2.15.1`. ([pantsbuild#19042](pantsbuild#19042))

* Fixup welcome-newcomers.yaml ([pantsbuild#19039](pantsbuild#19039))
illicitonion added a commit to illicitonion/pants that referenced this pull request May 26, 2023
Internal changes:

* Fix TypedDict construction ([pantsbuild#19166](pantsbuild#19166))

* Have package_dists.InvalidEntryPoint subclass InvalidFieldException ([pantsbuild#19149](pantsbuild#19149))

* Check for `CI=true`, not `CI=1` for cherry-picking ([pantsbuild#19141](pantsbuild#19141))

* Set up remote caching in CI  ([pantsbuild#19144](pantsbuild#19144))

* refactor(internal): adjust pants_venv venv_dir calculation ([pantsbuild#19140](pantsbuild#19140))

* Prepare `2.16.0rc4`. ([pantsbuild#19146](pantsbuild#19146))

* Warn about `files()` in PEXes in one place ([pantsbuild#19027](pantsbuild#19027))

* Drop our explicit dep on `syn` ([pantsbuild#19134](pantsbuild#19134))

* Fix auto-cherry-picker's merge-conflicts ([pantsbuild#19125](pantsbuild#19125))

* Use explicit depths on fetches when running cherry-picks in CI ([pantsbuild#19127](pantsbuild#19127))

* Make determine_python() pin to 3.9 ([pantsbuild#19116](pantsbuild#19116))

* Upgrade to `actions/checkout@v3` ([pantsbuild#19112](pantsbuild#19112))

* Run the cherry pick script with `-x` for debugging ([pantsbuild#19095](pantsbuild#19095))

* Add more linux x86 test CI shards ([pantsbuild#19099](pantsbuild#19099))

* Use a trait for remote ByteStore network implementation ([pantsbuild#19050](pantsbuild#19050))

* Increase the logging of release version file publishing. ([pantsbuild#19092](pantsbuild#19092))

* Remove the Toolchain plugin ([pantsbuild#18978](pantsbuild#18978))

* Fix bad merge of pantsbuild#18829. ([pantsbuild#19080](pantsbuild#19080))

* Revert "Implement layout="zip" for Lambda/GCF, skipping lambdex" ([pantsbuild#19071](pantsbuild#19071))

* Remove unnecessary interpreter constraint config. ([pantsbuild#19065](pantsbuild#19065))

* Use hand-written change detection for PR/push job trimming ([pantsbuild#19045](pantsbuild#19045))

* Do not build wheels on push, since that is now accomplished by the release job. ([pantsbuild#19063](pantsbuild#19063))

* Additional release job publishing fixes ([pantsbuild#19058](pantsbuild#19058))

* Change CI trigger to includelist `main` and `2.*.x` ([pantsbuild#19059](pantsbuild#19059))

* Release doc and script fixes ([pantsbuild#19043](pantsbuild#19043))

* Update cherry-picker to use `pull_request_target` ([pantsbuild#19048](pantsbuild#19048))

* Prepare `2.16.0rc3`. ([pantsbuild#19044](pantsbuild#19044))

* Prepare `2.15.1`. ([pantsbuild#19042](pantsbuild#19042))

* Fixup welcome-newcomers.yaml ([pantsbuild#19039](pantsbuild#19039))
illicitonion added a commit to illicitonion/pants that referenced this pull request May 26, 2023
Internal changes:

* Fix TypedDict construction ([pantsbuild#19166](pantsbuild#19166))

* Have package_dists.InvalidEntryPoint subclass InvalidFieldException ([pantsbuild#19149](pantsbuild#19149))

* Check for `CI=true`, not `CI=1` for cherry-picking ([pantsbuild#19141](pantsbuild#19141))

* Set up remote caching in CI  ([pantsbuild#19144](pantsbuild#19144))

* refactor(internal): adjust pants_venv venv_dir calculation ([pantsbuild#19140](pantsbuild#19140))

* Prepare `2.16.0rc4`. ([pantsbuild#19146](pantsbuild#19146))

* Warn about `files()` in PEXes in one place ([pantsbuild#19027](pantsbuild#19027))

* Drop our explicit dep on `syn` ([pantsbuild#19134](pantsbuild#19134))

* Fix auto-cherry-picker's merge-conflicts ([pantsbuild#19125](pantsbuild#19125))

* Use explicit depths on fetches when running cherry-picks in CI ([pantsbuild#19127](pantsbuild#19127))

* Make determine_python() pin to 3.9 ([pantsbuild#19116](pantsbuild#19116))

* Upgrade to `actions/checkout@v3` ([pantsbuild#19112](pantsbuild#19112))

* Run the cherry pick script with `-x` for debugging ([pantsbuild#19095](pantsbuild#19095))

* Add more linux x86 test CI shards ([pantsbuild#19099](pantsbuild#19099))

* Use a trait for remote ByteStore network implementation ([pantsbuild#19050](pantsbuild#19050))

* Increase the logging of release version file publishing. ([pantsbuild#19092](pantsbuild#19092))

* Remove the Toolchain plugin ([pantsbuild#18978](pantsbuild#18978))

* Fix bad merge of pantsbuild#18829. ([pantsbuild#19080](pantsbuild#19080))

* Revert "Implement layout="zip" for Lambda/GCF, skipping lambdex" ([pantsbuild#19071](pantsbuild#19071))

* Remove unnecessary interpreter constraint config. ([pantsbuild#19065](pantsbuild#19065))

* Use hand-written change detection for PR/push job trimming ([pantsbuild#19045](pantsbuild#19045))

* Do not build wheels on push, since that is now accomplished by the release job. ([pantsbuild#19063](pantsbuild#19063))

* Additional release job publishing fixes ([pantsbuild#19058](pantsbuild#19058))

* Change CI trigger to includelist `main` and `2.*.x` ([pantsbuild#19059](pantsbuild#19059))

* Release doc and script fixes ([pantsbuild#19043](pantsbuild#19043))

* Update cherry-picker to use `pull_request_target` ([pantsbuild#19048](pantsbuild#19048))

* Prepare `2.16.0rc3`. ([pantsbuild#19044](pantsbuild#19044))

* Prepare `2.15.1`. ([pantsbuild#19042](pantsbuild#19042))

* Fixup welcome-newcomers.yaml ([pantsbuild#19039](pantsbuild#19039))
illicitonion added a commit to illicitonion/pants that referenced this pull request May 26, 2023
Internal changes:

* Fix TypedDict construction ([pantsbuild#19166](pantsbuild#19166))

* Have package_dists.InvalidEntryPoint subclass InvalidFieldException ([pantsbuild#19149](pantsbuild#19149))

* Check for `CI=true`, not `CI=1` for cherry-picking ([pantsbuild#19141](pantsbuild#19141))

* Set up remote caching in CI  ([pantsbuild#19144](pantsbuild#19144))

* refactor(internal): adjust pants_venv venv_dir calculation ([pantsbuild#19140](pantsbuild#19140))

* Prepare `2.16.0rc4`. ([pantsbuild#19146](pantsbuild#19146))

* Warn about `files()` in PEXes in one place ([pantsbuild#19027](pantsbuild#19027))

* Drop our explicit dep on `syn` ([pantsbuild#19134](pantsbuild#19134))

* Fix auto-cherry-picker's merge-conflicts ([pantsbuild#19125](pantsbuild#19125))

* Use explicit depths on fetches when running cherry-picks in CI ([pantsbuild#19127](pantsbuild#19127))

* Make determine_python() pin to 3.9 ([pantsbuild#19116](pantsbuild#19116))

* Upgrade to `actions/checkout@v3` ([pantsbuild#19112](pantsbuild#19112))

* Run the cherry pick script with `-x` for debugging ([pantsbuild#19095](pantsbuild#19095))

* Add more linux x86 test CI shards ([pantsbuild#19099](pantsbuild#19099))

* Use a trait for remote ByteStore network implementation ([pantsbuild#19050](pantsbuild#19050))

* Increase the logging of release version file publishing. ([pantsbuild#19092](pantsbuild#19092))

* Remove the Toolchain plugin ([pantsbuild#18978](pantsbuild#18978))

* Fix bad merge of pantsbuild#18829. ([pantsbuild#19080](pantsbuild#19080))

* Revert "Implement layout="zip" for Lambda/GCF, skipping lambdex" ([pantsbuild#19071](pantsbuild#19071))

* Remove unnecessary interpreter constraint config. ([pantsbuild#19065](pantsbuild#19065))

* Use hand-written change detection for PR/push job trimming ([pantsbuild#19045](pantsbuild#19045))

* Do not build wheels on push, since that is now accomplished by the release job. ([pantsbuild#19063](pantsbuild#19063))

* Additional release job publishing fixes ([pantsbuild#19058](pantsbuild#19058))

* Change CI trigger to includelist `main` and `2.*.x` ([pantsbuild#19059](pantsbuild#19059))

* Release doc and script fixes ([pantsbuild#19043](pantsbuild#19043))

* Update cherry-picker to use `pull_request_target` ([pantsbuild#19048](pantsbuild#19048))

* Prepare `2.16.0rc3`. ([pantsbuild#19044](pantsbuild#19044))

* Prepare `2.15.1`. ([pantsbuild#19042](pantsbuild#19042))

* Fixup welcome-newcomers.yaml ([pantsbuild#19039](pantsbuild#19039))
illicitonion added a commit to illicitonion/pants that referenced this pull request May 26, 2023
Internal changes:

* Fix TypedDict construction ([pantsbuild#19166](pantsbuild#19166))

* Have package_dists.InvalidEntryPoint subclass InvalidFieldException ([pantsbuild#19149](pantsbuild#19149))

* Check for `CI=true`, not `CI=1` for cherry-picking ([pantsbuild#19141](pantsbuild#19141))

* Set up remote caching in CI  ([pantsbuild#19144](pantsbuild#19144))

* refactor(internal): adjust pants_venv venv_dir calculation ([pantsbuild#19140](pantsbuild#19140))

* Prepare `2.16.0rc4`. ([pantsbuild#19146](pantsbuild#19146))

* Warn about `files()` in PEXes in one place ([pantsbuild#19027](pantsbuild#19027))

* Drop our explicit dep on `syn` ([pantsbuild#19134](pantsbuild#19134))

* Fix auto-cherry-picker's merge-conflicts ([pantsbuild#19125](pantsbuild#19125))

* Use explicit depths on fetches when running cherry-picks in CI ([pantsbuild#19127](pantsbuild#19127))

* Make determine_python() pin to 3.9 ([pantsbuild#19116](pantsbuild#19116))

* Upgrade to `actions/checkout@v3` ([pantsbuild#19112](pantsbuild#19112))

* Run the cherry pick script with `-x` for debugging ([pantsbuild#19095](pantsbuild#19095))

* Add more linux x86 test CI shards ([pantsbuild#19099](pantsbuild#19099))

* Use a trait for remote ByteStore network implementation ([pantsbuild#19050](pantsbuild#19050))

* Increase the logging of release version file publishing. ([pantsbuild#19092](pantsbuild#19092))

* Remove the Toolchain plugin ([pantsbuild#18978](pantsbuild#18978))

* Fix bad merge of pantsbuild#18829. ([pantsbuild#19080](pantsbuild#19080))

* Revert "Implement layout="zip" for Lambda/GCF, skipping lambdex" ([pantsbuild#19071](pantsbuild#19071))

* Remove unnecessary interpreter constraint config. ([pantsbuild#19065](pantsbuild#19065))

* Use hand-written change detection for PR/push job trimming ([pantsbuild#19045](pantsbuild#19045))

* Do not build wheels on push, since that is now accomplished by the release job. ([pantsbuild#19063](pantsbuild#19063))

* Additional release job publishing fixes ([pantsbuild#19058](pantsbuild#19058))

* Change CI trigger to includelist `main` and `2.*.x` ([pantsbuild#19059](pantsbuild#19059))

* Release doc and script fixes ([pantsbuild#19043](pantsbuild#19043))

* Update cherry-picker to use `pull_request_target` ([pantsbuild#19048](pantsbuild#19048))

* Prepare `2.16.0rc3`. ([pantsbuild#19044](pantsbuild#19044))

* Prepare `2.15.1`. ([pantsbuild#19042](pantsbuild#19042))

* Fixup welcome-newcomers.yaml ([pantsbuild#19039](pantsbuild#19039))
illicitonion added a commit that referenced this pull request May 27, 2023
Internal changes:

* Fix TypedDict construction
([#19166](#19166))

* Have package_dists.InvalidEntryPoint subclass InvalidFieldException
([#19149](#19149))

* Check for `CI=true`, not `CI=1` for cherry-picking
([#19141](#19141))

* Set up remote caching in CI
([#19144](#19144))

* refactor(internal): adjust pants_venv venv_dir calculation
([#19140](#19140))

* Prepare `2.16.0rc4`.
([#19146](#19146))

* Warn about `files()` in PEXes in one place
([#19027](#19027))

* Drop our explicit dep on `syn`
([#19134](#19134))

* Fix auto-cherry-picker's merge-conflicts
([#19125](#19125))

* Use explicit depths on fetches when running cherry-picks in CI
([#19127](#19127))

* Make determine_python() pin to 3.9
([#19116](#19116))

* Upgrade to `actions/checkout@v3`
([#19112](#19112))

* Run the cherry pick script with `-x` for debugging
([#19095](#19095))

* Add more linux x86 test CI shards
([#19099](#19099))

* Use a trait for remote ByteStore network implementation
([#19050](#19050))

* Increase the logging of release version file publishing.
([#19092](#19092))

* Remove the Toolchain plugin
([#18978](#18978))

* Fix bad merge of #18829.
([#19080](#19080))

* Revert "Implement layout="zip" for Lambda/GCF, skipping lambdex"
([#19071](#19071))

* Remove unnecessary interpreter constraint config.
([#19065](#19065))

* Use hand-written change detection for PR/push job trimming
([#19045](#19045))

* Do not build wheels on push, since that is now accomplished by the
release job. ([#19063](#19063))

* Additional release job publishing fixes
([#19058](#19058))

* Change CI trigger to includelist `main` and `2.*.x`
([#19059](#19059))

* Release doc and script fixes
([#19043](#19043))

* Update cherry-picker to use `pull_request_target`
([#19048](#19048))

* Prepare `2.16.0rc3`.
([#19044](#19044))

* Prepare `2.15.1`.
([#19042](#19042))

* Fixup welcome-newcomers.yaml
([#19039](#19039))
huonw added a commit that referenced this pull request Jun 1, 2023
This separates the `remote::remote_cache` coordination code from the
gRPC REAPI implementation by:

- adding a `remote::remote_cache::ActionCacheProvider` trait
- moving the REAPI implementation into `remote::remote_cache::reapi` and
implementing that trait

This is, in theory, just a lift-and-shift, with no functionality change.

This is preparation work for supporting more remote stores like GHA
cache or S3, for #11149, and
is specifically broken out of
#17840. It continues #19050.
Additional work required to actually solve
#11149:

- implementing other byte store and action cache providers
- dynamically choosing the right providers for
`store::remote::ByteStore` and `remote::remote_cache`

The commits are individually reviewable:
1. preparatory breaking out of gRPC code
2. defining the trait
3. move the REAPI code and implement the trait, close to naively as
possible:
  - https://gist.github.com/huonw/a60ad807b05ecea98387294c22de67cb has a
    white-space-ignoring diff between `remote_cache.rs` after 1, and the
    `reapi.rs` file in this commit (it's less useful than #19050, since most
    of the code is deleted, but, buried in there are chunks of completely
    unchanged code)
4. minor clean-up
stuhood pushed a commit that referenced this pull request Sep 7, 2023
This fixes #19732 by restoring the retries when storing hits retryable
server failures from the REAPI remote cache server, which were lost in
the #19050 refactoring.

This also explicitly tests for retries, refactoring `StubCAS` to
generalise `read_request_count` to expose the counts of more requests
than just reads, and also consistently return a `Status::internal(...)`
for the simulated errors.

I think #19050 fortunately landed just after 2.17 was cut, so this
regression only affects the 2.18 pre-releases.
WorkerPants pushed a commit that referenced this pull request Sep 7, 2023
This fixes #19732 by restoring the retries when storing hits retryable
server failures from the REAPI remote cache server, which were lost in
the #19050 refactoring.

This also explicitly tests for retries, refactoring `StubCAS` to
generalise `read_request_count` to expose the counts of more requests
than just reads, and also consistently return a `Status::internal(...)`
for the simulated errors.

I think #19050 fortunately landed just after 2.17 was cut, so this
regression only affects the 2.18 pre-releases.
huonw added a commit that referenced this pull request Sep 7, 2023
…#19737) (#19798)

This fixes #19732 by restoring the retries when storing hits retryable
server failures from the REAPI remote cache server, which were lost in
the #19050 refactoring.

This also explicitly tests for retries, refactoring `StubCAS` to
generalise `read_request_count` to expose the counts of more requests
than just reads, and also consistently return a `Status::internal(...)`
for the simulated errors.

I think #19050 fortunately landed just after 2.17 was cut, so this
regression only affects the 2.18 pre-releases.

Co-authored-by: Huon Wilson <huon@exoflare.io>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

category:internal CI, fixes for not-yet-released features, etc.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants