Skip to content

fix(engine): a no-op resource keeps stale downstream edges, so destroy ordering races #979

Description

@aryasaatvik

A resource whose plan action is noop never re-persists its downstream edges, so any dependent added to (or removed from) it in that deploy is invisible to later dependency ordering. Destroy then tears the resource down concurrently with something that still depends on it.

Repro

Surfaced by a live SES test (#976), but nothing about it is SES-specific.

// deploy 1
const tenant = yield* Tenant("Customer", {});
const first = yield* ConfigurationSet("ConfigA", {});
const second = yield* ConfigurationSet("ConfigB", {});
yield* TenantResourceAssociation("Link", {
  tenantName: tenant.tenantName,
  resourceArn: first.configurationSetArn,   // Link -> ConfigA
});

// deploy 2 — same stack, association re-pointed. `resourceArn` is immutable,
// so Link is REPLACED. Both config sets are unchanged.
yield* TenantResourceAssociation("Link", {
  tenantName: tenant.tenantName,
  resourceArn: second.configurationSetArn,  // Link -> ConfigB
});

// deploy 2 plan: [ConfigA] noop  [ConfigB] noop  [Link] replace

Destroy:

[Link] deleting            ← concurrent
[ConfigB] deleting
[ConfigB] fail: BadRequestException: Cannot delete <arn> because it has
                tenant associations. Remove all tenant associations and try again.
[Link] deleted
[ConfigA] deleting         ← correctly waited for Link
[ConfigA] deleted

ConfigA — which Link no longer depends on — was ordered after Link. ConfigB — which it actually depends on — raced it.

Cause

Destroy ordering reads the downstream edges persisted on each resource (Plan.ts:539):

const oldDownstreamDependencies = Object.fromEntries(
  oldResources.filter(Boolean).map((r) => [r.fqn, r.downstream]),
);

Create, update, and replace all write downstream: node.downstream when they commit (Apply.ts 537 / 561 / 598 / 648). The noop branch does not (Apply.ts:503):

if (node.action === "noop") {
  if (node.state.resourceType !== node.resource.Type) {
    yield* commit({ ...node.state, resourceType: node.resource.Type });
  }
  yield* signalReadyStable;
  yield* storeAndSignal({
    output: node.state.attr,
    props: node.state.props,
    bindings: node.state.bindings ?? [],
    instanceId: node.state.instanceId,
  });
  return;   // node.downstream is never persisted
}

So an unchanged resource keeps whatever edge list was current the last time it was itself created or updated. In the repro ConfigB still carries the empty list from deploy 1, and ConfigA still carries [Link].

The freshly planned value is already on the node — node.downstream is what every other branch writes — so the edges are available, just not committed.

Impact

Silent wherever the parent API tolerates deleting while children exist, and an intermittent destroy failure wherever it doesn't — security groups with attached ENIs, VPCs with subnets, buckets with objects, configuration sets with associations. It needs only a dependent to be added to or removed from an otherwise-unchanged resource, which makes "re-point a child at a different parent" the common trigger. The failure looks provider-specific, so it tends to get papered over in the provider or the test rather than traced here.

Related: ResourceState.ts:86 carries a TODO(sam): do I need to track the old downstream edges? on UpdatingResourceState.old — same area, different question.

Suggested fix

Persist node.downstream in the noop branch. A no-op means the resource needs no API call; it does not mean its graph position is unchanged.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions