Maintenance: e2e jobs intermittently fail with toolkit-lib "NoStack" during deploy

[svozza]

Summary

Multiple e2e matrix jobs fail intermittently with a toolkit-lib error during stack deploy:

NoStack: ❌  <StackName> failed: NoStack: CloudFormationStack object does not hold a stack

This reproduces on a clean main — run 31155239957 (commit ac38354c) had 6 of 40 cells fail this way across unrelated packages (idempotency ×2, logger, event-handler, tracer, parameters), so it is not specific to any one suite or PR. It also appeared on a feature branch at similar rates, independent of the changes there.

CloudTrail for the failing stacks shows a consistent signature: CreateChangeSet succeeds, ExecuteChangeSet succeeds (no errorCode), and ~30s later the toolkit throws NoStack — i.e. the change set executes, but a follow-up DescribeStacks reads the stack back as absent.

Tracing @aws-cdk/toolkit-lib 1.35.0: in deploy-stack.js (monitorDeployment), waitForStackDeploy → stabilizeStack polls DescribeStacks after execute; when that read comes back empty (a read-after-write consistency window on a just-created stack), it throws a DeploymentError. The catch then calls finalState.wrapped to diagnose it, but finalState is the pre-deploy CloudFormationStack (empty for a brand-new stack), so .wrapped throws NoStack (cloudformation/stack-helpers.js:58) — masking the underlying stabilization failure. This is the same class of CloudFormation read-after-write / eventual-consistency sensitivity as the polling/throttling issues previously raised upstream (aws/aws-cdk-cli#1709, aws/aws-cdk-cli#1723); it surfaces under the high CFN concurrency of our matrix.

Why is this needed?

6/40 cells red on a clean main run means most e2e runs need a "re-run failed jobs" pass, which wastes maintainer time, burns CI/AWS resources, and erodes trust in the signal (real failures hide among the flakes). It also compounds with the LMI/throttling work already in flight.

Which area does this relate to?

Tests, Automation

Solution

We have several options available but my preference is to fix upstream:

• Make the TestStack deploy tolerant of this transient: catch the toolkit NoStack/stabilization error and retry the deploy a small number of times (the stack usually becomes consistent within seconds). Lowest-effort, contained to packages/testing.
• Raise upstream with aws-cdk-cli: the post-ExecuteChangeSet lookup in stabilizeStack/monitorDeployment should tolerate an eventual-consistency window rather than treating an empty DescribeStacks as terminal, and the error-diagnosis path should not mask the real error with NoStack. (Sibling to Maintenance: build Lambda Layers from source during tests #1709/feat(idempotency): add idempotency decorator #1723.)
• Investigate whether raising stackEventPollingInterval / the stabilization interval reduces the frequency.

Acknowledgment

• This request meets Powertools for AWS Lambda (TypeScript) Tenets
• Should this be considered in other Powertools for AWS Lambda languages? i.e. Python, Java, and .NET

[dreamorosi]

Thanks for the report.

Between all these nags, and the endless string of CVEs in transitive dependencies, I'm seriously starting to consider ditching CDK entirely and make the stacks deploy via AWS CLI directly, using YAML stacks.

With that said, my preference would be the second, then the first, and finally the third option - but I'm open to your ordering as well.

[svozza]

Yeah, actually my preference is for two as well, I meant that to be an unordered list. Also, agreed on the CDK, definitely open to removing it: we are getting into more hassle than it's worth territory.

[svozza]

Root-caused and fixed upstream: aws/aws-cdk-cli#1803 (fixes aws/aws-cdk-cli#1802).

The upstream diagnosis matches what we saw here and pins down the exact mechanism: DescribeStacks is eventually consistent, so a poll issued right after ExecuteChangeSet can still return the pre-execution REVIEW_IN_PROGRESS status. StackStatus.isInProgress excludes review, so stabilizeStack treated that stale read as stable and waitForStackDeploy rejected the deployment — while CloudFormation went on to reach CREATE_COMPLETE. The error was then masked as NoStack in the diagnosis path (the pre-deploy CloudFormationStack is empty for a fresh create). The fix keeps polling when a stale REVIEW_IN_PROGRESS is seen for a stack whose execution we've already issued, identified by stack id so a genuine concurrent same-name review isn't mistaken for a stale read.

Once aws/aws-cdk-cli#1803 ships in a @aws-cdk/toolkit-lib release, bumping the pin here should clear these flakes. This is also the blocker for merging #5465.