fix(diy-backend): tolerate 404 wrap regression in stack-checkpoint lookup

[Cre-eD]

Summary

External SC consumers on 2026.5.31 are seeing a hard failure on the first deploy of a parent stack:

deployment failed: failed to get parent stack "wize-rooms-api":
failed to get stack "wize-rooms-api":
failed to load checkpoint: blob (key ".pulumi/stacks/demo/wize-rooms-api.json") (code=Unknown):
storage: object doesn't exist: googleapi: Error 404: No such object:
likeclaw-simple-container-state/.pulumi/stacks/demo/wize-rooms-api.json, notFound

Pinning back to 2026.5.13 works around it, but blocks adopting any newer release (including #280 which is what PAY-SPACE needs for VPA controlledValues).

Root cause

A 404 from the state backend used to round-trip through gocloud's Bucket.Exists as (false, nil). Pulumi's diy backend mapped that to errCheckpointNotFound, GetStack returned (nil, nil), and SC's createStackIfNotExists would create the missing parent stack. That contract held through 2026.5.13.

The transitive bumps in #279 — most notably cloud.google.com/go/storage v1.49.0 → v1.62.2 alongside pkg/v3 v3.184.0 → v3.241.0 — change the error path so some GCS 404s reach gcerrors.Code as Unknown rather than NotFound. Bucket.Exists then returns (false, wrapped-err), stackExists wraps as "failed to load checkpoint", and GetStack returns the wrap — selectStack propagates and the deploy fails before createStackIfNotExists runs.

The diff in Pulumi's diy backend (GetStack, stackExists, errCheckpointNotFound) between v3.184.0 and v3.241.0 is byte-identical — the regression is entirely in the transitive cloud-storage client.

Fix

In selectStack, detect 404-shaped errors from the diy backend's "failed to load checkpoint:" wrap and treat them the same as the (nil, nil) return that v3.184 produced.

• Primary: gcerrors.Code(err) == gcerrors.NotFound (the structured path; handles future-fixed clients).
• Fallback: scoped string match for "failed to load checkpoint:" + (NotFound marker) across GCS / S3 / Azure provider 404 shapes. Scoped to the checkpoint wrapper to avoid swallowing unrelated NotFound-shaped errors.

Test plan

• go test ./pkg/clouds/pulumi/... — passes locally.
• go build ./... — passes locally.
• New unit test TestStackCheckpointNotFound:
  • nil → false
  • GCS 404 wrapped through Pulumi diy backend (the exact customer error string) → true
  • GCS 404 without the checkpoint prefix → false (out of scope)
  • S3 NoSuchKey wrapped → true
  • Azure BlobNotFound wrapped → true
  • "failed to load checkpoint: permission denied" → false (not a NotFound shape)

Why a string fallback

The structured path is preferred — gcerrors.Code is the documented surface. But the regression bites precisely because the cloud-storage client's wrap stopped classifying as NotFound. Defending only at the structured layer leaves the regression in place. A scoped string check on a known marker is the smallest patch that restores the contract until upstream gocloud / cloud-storage settles.

Why not just downgrade the storage dep

Could be argued, but:

1. We'd lose security fixes that came with the bump.
2. The transitive-dep landscape is unstable enough that another bump could re-trigger the same regression in a different shape.
3. Defensive normalization at the SC seam matches the contract we already advertise to createStackIfNotExists.

Refs: external consumer report 2026-05-21T12:42:35Z, #279, #280

[github-actions]

Semgrep Scan Results

Repository: api | Commit: 6a655eb

Check	Status	Details
⚠️ Semgrep	Warning	10 warning(s), 10 total

Scanned at 2026-05-21 17:15 UTC

[github-actions]

Security Scan Results

Repository: api | Commit: 6a655eb

Check	Status	Details
✅ Secret Scan	Pass	No secrets detected
✅ Dependencies (Trivy)	Pass	0 total (no critical/high)
✅ Dependencies (Grype)	Pass	0 total (no critical/high)
📦 SBOM	Generated	528 components (CycloneDX)

Scanned at 2026-05-21 17:15 UTC

[Cre-eD]

Reviewed by Codex + Gemini (independent runs).

Codex

The GCS-specific path appears covered, but the fallback is too narrow for a documented S3 missing-object 404 format.

[P2] For S3 v2 SDK errors that surface as api error NotFound: Not Found / StatusCode: 404, the original marker list (NoSuchKey + lowercase notFound) wouldn't fire.

Gemini

[P0] String matching is brittle and case-sensitive — vulnerable to upstream formatting drift ("notFound" vs "Not Found" vs "NotFound").

[P3] Stronger alternative: errors.As(err, &googleapi.Error{}) typed-error path.

Both points addressed in follow-up commit 806642c:

Change	Why
strings.ToLower(msg) + lowercase markers	Defends against case-drift
Added statuscode: 404 marker	Catches AWS SDK v2 HeadObject 404 wraps + future SDKs
Added error 404 marker	Matches the customer's exact googleapi: Error 404 wrap
2 new test cases (S3 v2, case-insensitive, generic 404)	Locks in coverage

Style note (not addressed)

• Gemini P3: typed errors via errors.As would be more robust but means coupling SC to provider-specific error types (*googleapi.Error, AWS SDK v2 *types.NoSuchKey, Azure *azcore.ResponseError). Worth opening as a follow-up if the string fallback ever misfires; for the current regression the broadened matcher is sufficient and self-contained.

Docs

Followup commit 116cd6f documents the controlledValues field added in #280 — flagged during this review pass since the doc gap was noticed while explaining the matcher.

Final tests

go build ./...                                       ✓
go vet ./pkg/clouds/pulumi/...                       ✓
go test ./pkg/clouds/pulumi/... -count=1             ✓
9/9 unit tests pass in TestStackCheckpointNotFound:
  nil, GCS, GCS-no-prefix, S3 v1 NoSuchKey,
  S3 v2 NotFound, Azure BlobNotFound,
  case-insensitivity guard, generic StatusCode: 404,
  permission-denied (negative)