Add buildcache channel for Build Caching Service runtime resolution

[LoopedBard3]

Adds a new buildcache channel that resolves .NET runtime binaries from the Build Caching Service (BCS) instead of NuGet feeds. This gives per-commit granularity for performance regression bisection, instead of being limited to feed cadence.

Wire-up

Controller (Job.cs) — new opt-in properties (default ""):

• BuildCacheCommitSha — pin a specific commit; empty = latest from BCS
• BuildCacheBranch — default main
• BuildCacheConfig — defaults to the inferred RID config (e.g. coreclr_x64_linux)

Agent (Startup.cs / BuildCacheClient.cs) — new CLI:

• --build-cache-base-url (default: https://pvscmdupload.z22.web.core.windows.net)
• --build-cache-repo-name (default: runtime)
• --build-cache-disabled

Usage:

--application.channel buildcache \
    [--application.buildCacheCommitSha <full-or-prefix-sha>] \
    [--application.buildCacheBranch main] \
    [--application.buildCacheConfig coreclr_x64_linux]

How runtime resolution works

The agent still builds the app with real NuGet packages (so dotnet restore/publish behave normally and runtimeconfig.json references a real installed version), then swaps in BCS bits at the right layer depending on publish kind:

• Self-contained: BCS runtime files are overlaid onto the published outputFolder. The SDK-bound apphost is preserved — BCS ships an unbound apphost containing a SHA-256 placeholder where the managed DLL path is encoded, so overlaying it would break startup.
• Framework-dependent: a per-job isolated dotnet home is created at <temp>/crank-buildcache/home-<shortsha>-<config>-<guid>/, mirrored from the global _dotnethome and overlaid with BCS files. StartProcess resolves the runtime from this isolated home. The global _dotnethome is never mutated, so concurrent jobs and other channels are unaffected.

The resolved BCS commit is recorded in MeasurementMetadata (under runtime-version) so reports show the real binary that ran, not the feed-resolved string.

Cache correctness

The BCS commit, branch, and config are included in BuildKeyData, so --application.options.reuseBuild correctly cache-busts when any of them change. Pre-existing options.json files written by older agents will cache-miss exactly once after upgrade, then re-cache in the new format.

Backward compatibility

All new code is gated on --application.channel buildcache. Old controllers calling a new agent are unaffected — new BuildCache* properties default to "" and are read defensively (!string.IsNullOrEmpty(...)); the BCS path is never entered without the opt-in channel.

Verification

• dotnet build clean
• 36 unit tests in test/Microsoft.Crank.UnitTests/BuildCacheClientTests.cs cover ParseLatestBuilds (multiple JSON shapes), ValidateCommitSha (incl. path-traversal rejection), OverlayPublishedOutput (incl. SDK-apphost preservation), CreateBuildCacheDotnetHome (incl. isolation between two concurrent jobs), SelectHighestManagedDir (numeric-aware: net11.0 > net10.0 > net9.0), CleanupExtractDir, etc.
• End-to-end on Linux (WSL agent, plaintext platform, 1024 connections): 8,456,046 RPS, metadata reports 11.0.0-preview.6.26277.104+38d408d22a64.
• End-to-end on Windows (hello sample): 162,667 RPS, metadata reports 11.0.0-preview.6.26277.116+c56e0a2499fa, isolated dotnet home confirmed.

[Copilot]

Pull request overview

This PR adds a new buildcache channel to Crank’s agent/runtime resolution pipeline so the runtime binaries can be sourced per-commit from the Build Caching Service (BCS), enabling finer-grained performance regression bisection than VMR feed cadence allows.

Changes:

• Added build-cache-related properties to Job for selecting commit/branch/config.
• Added BuildCacheClient for resolving latest builds, downloading/extracting artifacts, and overlaying runtime binaries.
• Updated agent startup/version resolution logic and docs to support the new buildcache channel and new agent CLI options.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 9 comments.

Show a summary per file

File	Description
src/Microsoft.Crank.Models/Job.cs	Adds job-level inputs for BCS commit/branch/config selection.
src/Microsoft.Crank.Agent/Startup.cs	Adds CLI options and integrates buildcache into runtime version resolution + post-publish overlay.
src/Microsoft.Crank.Agent/BuildCacheClient.cs	Implements BCS querying/downloading/extraction and overlay helpers.
docs/dotnet_versions.md	Documents buildcache channel usage and configuration knobs.
docs/build_cache_requirements.md	Describes required BCS-side blob layout/access for the integration.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

runtimeVersion is being mutated to include +buildcache.<sha> and then passed into PatchRuntimeConfig. For framework-dependent publishes (i.e., when runtimeOptions does not contain includedFrameworks), this will write a runtime version that is not actually installed, which can prevent the app from starting. Keep the feed-resolved runtime version for runtimeconfig patching and store the BCS commit separately for reporting/metadata, or gate buildcache usage to self-contained publishes only.

[LoopedBard3]

Addressed by 1ecc239 (round 1) and 56657e2 (round 2): the BCS commit is no longer appended to runtimeVersion. PatchRuntimeConfig keeps the feed-resolved version (which is actually installed), and the BCS commit is reported separately via MeasurementMetadata (runtime-version). FDD startup is no longer broken by the rewrite.

[Copilot]

BuildCache overlay is applied to the published output, but for non-self-contained publishes the output typically won’t contain runtime binaries to replace (and PatchRuntimeConfig will still run). If buildcache is intended to work only with job.SelfContained, enforce that with a clear error message; otherwise, consider using the existing BuildCacheClient.InstallRuntimeFromBuildCacheAsync approach to overlay into dotnetHome for framework-dependent runs.

[LoopedBard3]

Addressed by 19f909e (round 3) with a different approach: instead of mutating the global _dotnethome (which would create cross-job pollution), the agent now creates a per-job isolated dotnet home at <temp>/crank-buildcache/home-<shortsha>-<config>-<guid>/, mirrored from the global home and overlaid with BCS bits. StartProcess uses jobContext.BuildCacheDotnetHome for runtime resolution. FDD works correctly and concurrent jobs are isolated. The original InstallRuntimeFromBuildCacheAsync method was dead code and was deleted in round 1.

[Copilot]

_latestBuildsCache and _installedBuildCacheRuntimes are static non-thread-safe collections. The agent can process multiple jobs concurrently, so concurrent accesses can corrupt these collections or throw. Use ConcurrentDictionary/ConcurrentHashSet (or locks) around these caches, and also include baseUrl in the latestBuilds cache key to avoid cross-contaminating entries when the base URL is overridden.

[LoopedBard3]

Addressed by 1ecc239 / 19f909e: _latestBuildsCache is now a ConcurrentDictionary keyed by $"{baseUrl}|{repoName}|{branch}" so the base URL is included and concurrent reads/writes are safe. _installedBuildCacheRuntimes was part of the dead InstallRuntimeFromBuildCacheAsync design and was removed.

[Copilot]

This doc’s example URLs use the blob endpoint with /$web/..., but the agent default base URL (and client URL construction) uses the static website endpoint (https://pvscmdupload.z22.web.core.windows.net) without /$web. Please align the documented URLs with what the agent actually calls (or clarify which base URL should be configured via --build-cache-base-url).

[LoopedBard3]

Addressed by 1ecc239 (round 1): the doc URLs in docs/build_cache_requirements.md now match what the agent actually calls — the static-website endpoint https://pvscmdupload.z22.web.core.windows.net/builds/... (no /$web/). The default --build-cache-base-url value is documented and matches.