[Operational Issue]: Source-build jobs failing with circular dependency in target

[ViktorHofer]

Error

/__w/1/s/eng/PublishSourceBuild.props(166,5): error MSB4006: There is a circular dependency in the target dependency graph involving target "CreatePrivateSourceBuiltArtifactsArchive". [/__w/1/s/.packages/BootstrapPackages/microsoft.dotnet.arcade.sdk/11.0.0-beta.26381.103/toolset/Publish.proj]

Links

https://dev.azure.com/dnceng/internal/_build/results?buildId=3046380&view=logs&j=0c2a1884-6715-550d-a904-b6e3cc3a6e36&t=c1386052-8b09-50ae-d87a-8553b08837d2

Root Cause

Changed with the upgrade to the .NET 11 Preview 7 SDK

Blocking

• This issue is blocking

Affected Releases

• .NET 10
• .NET 10 (SDK band build)
• .NET 11

[ViktorHofer]

Root cause of the sudden appearance: MSBuild server enabled by default (dotnet/sdk#55231)

Follow-up investigation on why this pattern started failing now. The MSBuild task recursion in eng/PublishSourceBuild.props has been self-referential for a long time, but it was silently satisfied from the results cache. It is not an MSBuild engine regression.

What changed

dotnet/sdk#55231 - "Enable MSBuild server by default in MSBuildForwardingApp" (commit 16e4424e7, merged 2026-07-14) flipped the default of DOTNET_CLI_USE_MSBUILD_SERVER from false to true in src/Cli/Microsoft.DotNet.Cli.Utils/MSBuildForwardingAppWithoutLogging.cs, so the CLI now exports MSBUILDUSESERVER=1 for every dotnet build / dotnet msbuild invocation.

It flowed into the VMR with 0a2e6315b05 (2026-07-15), which is an ancestor of the failing build's SDK but not of the last-good one.

Evidence

Comparing the binlogs of the last green build (3045766) and the first red build (3046380):

	Good (3045766)	Failing (3046380)
SDK	11.0.100-preview.6.26359.118	11.0.100-preview.7.26381.103
MSBuild	18.9.0.36018	18.10.0.38203
MSBUILDUSESERVER in env	unset	1
Publish.proj -> Execute project nodes	3 (3rd satisfied from cache in ~22 us)	2, then MSB4006

The exact MSBuild commit range used by the two builds (a47a64a9..eae54023, derived from the Microsoft.Build.dll informational versions) contains no changes at all under src/Build/BackEnd/Components/Caching, .../Scheduler, or .../BuildRequestEngine - those files are byte-identical. So the engine's cache/scheduling behavior did not change.

Mechanism

TargetBuilder.cs publishes the in-flight BuildResult into the node-local results cache only when the configuration has no result yet:

BuildResult existingBuildResult = resultsCache.GetResultsForConfiguration(_requestEntry.Request.ConfigurationId);
_buildResult = new BuildResult(entry.Request, existingBuildResult, null);
if (existingBuildResult == null)
{
    // Add this result so that if our project gets re-entered we won't rebuild any targets we have already built.
    resultsCache.AddResult(_buildResult);
}

That is what used to answer the re-entrant pass-3 request instantly (Execute was already recorded as built). With the persistent MSBuild server the config/results caches are no longer pristine per invocation, the re-entrant request misses the cache, reaches the scheduler, and SchedulableRequest detects the same configuration id in the request's own ancestor chain -> MSB4006.

Conclusion

The recursion was always latently invalid; enabling the MSBuild server merely stopped masking it. #8276 (guarding the recursion with IsPublishPass2) is the correct fix, and no MSBuild-side change is needed.

[ViktorHofer]

Correction / elaboration on the mechanism

Someone asked me to expand on "the config/results caches are no longer pristine per invocation". I tried to verify that claim and could not reproduce it, so I want to correct it rather than leave it standing.

What I actually tested

Using a reduced repro of the Publish.proj target graph (InitialTargets + AfterTargets="Execute" + the self-recursive MSBuild task) against the exact SDK from the failing build (11.0.100-preview.7.26381.103):

Scenario	Result
MSBUILDUSESERVER=0	passes (3rd request satisfied from cache)
MSBUILDUSESERVER=1, cold server	passes
MSBUILDUSESERVER=1, 4 consecutive builds against the same warm server	passes every time

So a persistent server does not, on its own, leave stale configuration/result entries that cause the miss. That sentence in my earlier comment was wrong.

What remains proven

1. MSBUILDUSESERVER=1 is present in the environment of the failing build and absent in the last-good build, and that flip comes from dotnet/sdk#55231 (flowed into the VMR by 0a2e6315b05, 2026-07-15).
2. MSBuild's src/Build/BackEnd/Components/{Caching,Scheduler,BuildRequestEngine} and TargetBuilder.cs are byte-identical between the MSBuild revisions in the two builds (a47a64a9..eae54023). There is no engine-side regression.
3. In the good build the re-entrant request appears as a third Publish.proj -> Execute project node that completes in ~22 us with Execute reported as already built - i.e. answered from the results cache. In the failing build that node does not exist; the request reached the scheduler, which found the same configuration id in the request's own ancestor chain and raised MSB4006.

Why the cache used to answer it

TargetBuilder registers the in-flight BuildResult into the node-local results cache, but only when that configuration has no result yet:

BuildResult existingBuildResult = resultsCache.GetResultsForConfiguration(_requestEntry.Request.ConfigurationId);
_buildResult = new BuildResult(entry.Request, existingBuildResult, null);
if (existingBuildResult == null)
{
    // Add this result so that if our project gets re-entered we won't rebuild any targets we have already built.
    resultsCache.AddResult(_buildResult);
}

Because pass 2's live result was registered this way, and Execute had already completed by the time the AfterTargets target ran, the re-entrant pass-3 request for the same configuration was satisfied locally in BuildRequestEngine and never reached the scheduler. That is the masking behaviour, and it is inherently fragile: it depends on the config having no pre-existing result on that node.

What is still unproven

Exactly how enabling the server flips that outcome. The two builds differ in node placement (pass 1 on node 6 / pass 2 on node 8 when it worked, versus node 6 / node 7 when it failed), which suggests scheduling and node assignment - not cache staleness - is the proximate difference, but I could not reproduce that at repro scale. It also remains possible that MSBUILDUSESERVER=1 is a correlate of the newer SDK rather than the cause.

A cheap way to settle it: re-run an affected SB_*_Offline_MsftSdk leg on the pre-fix commit with DOTNET_CLI_USE_MSBUILD_SERVER=0. If MSB4006 disappears, the server is confirmed as the trigger.

None of this changes the fix: the recursion in eng/PublishSourceBuild.props was always self-referential and relied on an incidental cache hit, so guarding it with IsPublishPass2 (#8276) is correct regardless of which change stopped masking it.

[ViktorHofer]

Correction: the MSBuild Server is not the cause — retracting my earlier comment

My earlier comment attributed this to DOTNET_CLI_USE_MSBUILD_SERVER defaulting to true (dotnet/sdk#55231). That was wrong, and I'm retracting it.

MSBUILDUSESERVER=1 is set in the failing build (and not in the passing one), but the server was never actually used: node reuse is disabled in CI, so the client falls back to a normal in-process build. The "MSBuild Server was requested but not used for this build: node reuse is disabled" diagnostic is itself brand new in the failing toolset (state="new" in the .xlf resources), which is why the passing build never printed it. Both builds ran serverless, so the server is a correlate of the newer SDK, not a cause.

What the binlog comparison actually shows

	passing (3045766)	failing (3046380)
SDK	11.0.100-preview.6.26359.118	11.0.100-preview.7.26381.103
Enabled loggers	BinaryLogger, CentralForwardingLogger x2, MSBuildLogger, ConsoleLogger	same + PipelinesLogger, MSBuildForwardingLogger, ConfigurableForwardingLogger
Publish.proj -> Execute project nodes	3	2, then MSB4006
re-entrant request	satisfied from the node-local results cache in 22 us	reached the scheduler -> MSB4006

Everything else I could compare is identical: the pass-1 / pass-2 target sequences, the global properties on the re-entrant MSBuild task (IsPublishPass2=true, AssetManifestFileName=...), node count, and ExcludeRestorePackageImports.

The only non-toolset delta is the new logger. It comes from the Arcade flow in the same window: eng/common/build.sh gained a --pipelinesLog switch that now defaults to on under --ci, and eng/common/tools.sh injects -logger:Microsoft.DotNet.ArcadeLogging.PipelinesLogger when it is set. I can prove the correlation but not causality — see below.

Why it used to "work"

src/msbuild/src/Build/BackEnd/Components/RequestBuilder/TargetBuilder.cs:

BuildResult existingBuildResult = resultsCache.GetResultsForConfiguration(_requestEntry.Request.ConfigurationId);
_buildResult = new BuildResult(entry.Request, existingBuildResult, null);
if (existingBuildResult == null)
{
    // Add this result so that if our project gets re-entered we won't rebuild any targets we have already built.
    resultsCache.AddResult(_buildResult);
}

The in-flight BuildResult is registered in the node-local results cache, and Execute already has a recorded result by the time the AfterTargets="Execute" target runs. So the self-referential request was answered by BuildRequestEngine.IssueBuildRequests -> ResultsCache.SatisfyRequest and never reached the scheduler. In the failing build it does reach the scheduler, which correctly rejects a request whose own configuration is already in its ancestor chain.

What I ruled out

• MSBuild engine changes. Across the msbuild range between the two toolsets, BackEnd/Components/Caching, Components/Scheduler, Components/BuildRequestEngine, TargetBuilder.cs, TargetEntry.cs, TaskHost.cs and src/Tasks/MSBuild.cs are byte-identical. The BuildRequestConfiguration.RequestedTargets serialization change is only consumed by ProjectCacheService/CacheSerialization, not by the request or cache hot path.
• A race. All 6 legs that pass /p:DotNetSourceOnlyPublishArtifacts=true failed in 3046380 and all 6 passed in 3045766, so it is deterministic per toolset.
• Local reproduction. A reduced repro of the exact pattern passes on both SDKs, with the server on and off, cold and warm, at /m:8 with filler projects, and with an explicit central -logger:. I could not make the cache lookup miss outside CI.

Bottom line

The recursion in eng/PublishSourceBuild.props was always an invalid self-referential build request; it only ever succeeded because the node-local results cache happened to answer it before the scheduler saw it. Something in the toolset/logging flow of that window stopped masking it. #8276 removes the recursion in pass 2, which is correct regardless of which change removed the masking.

To settle causality definitively, re-run an affected leg on the pre-fix commit with the PipelinesLogger suppressed (-p: / pipelines_log=false), or with MSBUILDDEBUGSCHEDULER=1 to capture the scheduler trace.

[ViktorHofer]

Root cause found and proven: a skipped InitialTarget added by Arcade

This is fully explained now. It is not the MSBuild Server and not the PipelinesLogger (both of my earlier comments are retracted). It is also not an MSBuild engine change - the failure reproduces identically on both the passing and failing toolsets.

The change

dotnet/arcade changed src/Microsoft.DotNet.Arcade.Sdk/tools/Workarounds.targets from:

<Project>

to:

<Project InitialTargets="NormalizeNetCoreSdkRootCasing">

It flowed into the VMR as 088599371eb ("[main] Source code updates from dotnet/arcade (#7360)", 2026-06-25). The target itself is a Windows-only drive-casing workaround:

<Target Name="NormalizeNetCoreSdkRootCasing"
        Condition="'$(MSBuildRuntimeType)' == 'Full' and '$(NetCoreSdkRoot)' != '' and ...">

Workarounds.targets is imported by the Arcade SDK for every Arcade project, and InitialTargets on an imported file aggregates into the importing project's InitialTargets. So Arcade's toolset/Publish.proj went from InitialTargets = DiscoverArtifacts to InitialTargets = DiscoverArtifacts;NormalizeNetCoreSdkRootCasing.

On Linux, MSBuildRuntimeType is Core, so the condition is false and the target is recorded with TargetResultCode.Skipped.

Why a skipped initial target breaks the build

ResultsCache.SatisfyRequest refuses to satisfy a request if any initial target has a skipped result:

// Check for the initial targets. If we don't know what the initial targets are, we assume they are not satisfied.
if (configInitialTargets == null || !CheckResults(allResults, configInitialTargets, checkTargetsMissingResults: false, skippedResultsDoNotCauseCacheMiss))
{
    response.Type = ResultsCacheResponseType.NotSatisfied;
}

and CheckResults bails on TargetResultCode.Skipped unless skippedResultsDoNotCauseCacheMiss is set, which only happens under project isolation:

internal bool SkippedResultsDoNotCauseCacheMiss() => ProjectIsolationMode == ProjectIsolationMode.True;

So the chain is:

1. CreatePrivateSourceBuiltArtifactsArchive (AfterTargets="Execute") re-invokes its own project with Targets="Execute" and the same global properties, i.e. the same configuration.
2. Previously that self-referential request was answered by the node-local results cache (BuildRequestEngine.IssueBuildRequests -> ResultsCache.SatisfyRequest), because TargetBuilder registers the in-flight BuildResult so a re-entered project does not rebuild targets it already built. In the passing binlog this shows up as a third Publish.proj -> Execute project node completing in 22 microseconds.
3. With NormalizeNetCoreSdkRootCasing now present and skipped, SatisfyRequest returns NotSatisfied.
4. The request therefore goes to the scheduler, which finds the request's own configuration in its ancestor chain and raises SchedulerCircularDependencyException -> MSB4006, reported at the <MSBuild> task's location in eng/PublishSourceBuild.props.

Deterministic repro (no CI needed)

Two projects, identical except for the initial targets; recursion one level below the entry project:

repro	InitialTargets	preview.6 SDK	preview.7 SDK
control	DiscoverArtifacts	passes	passes
skippedinit	DiscoverArtifacts;NormalizeCasing (condition false)	MSB4006	MSB4006

Same result on both toolsets, which confirms the MSBuild version is irrelevant.

Why this build and not the previous one

The bootstrap SDK/Arcade moved from 11.0.100-preview.6.26359.118 to 11.0.100-preview.7.26381.103. release/11.0.1xx-preview6 (ba53d0ed, 2026-07-09) branched from main before the Arcade change; release/11.0.1xx-preview7 (e2c1e00b, 2026-07-31) includes it. The binlogs confirm it directly: NormalizeNetCoreSdkRootCasing is absent from the passing binlog and present in the failing one.

It failed on all 6 legs that pass /p:DotNetSourceOnlyPublishArtifacts=true, deterministically, because the recursion only runs for those legs and the target is always skipped on Linux.

Notes

• This only breaks the in-flight re-entrant case. For a configuration that has already completed, a repeated request still avoids re-running targets via TargetBuilder's "already built" logic, so there is no general cache/perf regression - I verified this.
• The same latent bug was hit before and fixed the same way on release/10.0.2xx in Internal 2xx fixes #4932, which adds the identical and '$(IsPublishPass2)' != 'true' guard. Any skipped initial target on that configuration will expose it.
• Fix MSB4006 circular dependency in source-only publish pass 2 #8276 removes the invalid self-referential request, which is the correct fix independent of what removed the masking.