Summary
Add a scheduled GitHub Actions workflow that removes leaked e2e infrastructure from the test account.
This is a follow-up to #5465, which introduces run-scoped shared Lambda Managed Instances (LMI) capacity provider stacks named LmiShared-<runId>-<arch>. These stacks include EC2 fleets with a minimum capacity of 12 vCPUs and no scale-to-zero, a VPC, and a CloudWatch Logs interface endpoint. Unlike existing serverless e2e leftovers, leaked LMI infrastructure incurs hourly charges.
This issue tracks only the scheduled sweeper. Complementary mitigations, including job timeout-minutes and a destroy retry/second pass, are being handled during the review of #5465.
Why is this needed?
A cancelled or timed-out LMI test cell can leave its per-suite function stack running, with the function attached to the shared capacity provider by ARN. Because this relationship crosses stacks and is invisible to CloudFormation, the if: always() teardown job can encounter DELETE_FAILED when deleting the provider stack. The current destroy script does not retry, and no later process revisits leaked resources in the account.
Other leak paths include:
- The teardown job being blocked by runner availability.
- Environment protection rules preventing teardown from starting.
- Runs using
DISABLE_TEARDOWN.
LmiShared-local-* stacks created from developer machines.
A recurring cleanup mechanism is needed as a final safety net to limit cost and prevent abandoned e2e resources from accumulating.
This is a follow-up to #5465 and is related to #5092 and #5518.
Which area does this relate to?
Solution
Add a cron-based GitHub Actions workflow that runs nightly or every few hours and can also be invoked manually. It should assume the existing E2E OIDC role and require no standing AWS infrastructure of its own.
In eu-west-1, the workflow should:
- List CloudFormation stacks that are older than an age threshold of approximately three hours and either:
- Match
LmiShared-*, including LmiShared-local-*; or
- Have the tag
Service: Powertools-for-AWS-e2e-tests, which all e2e stacks receive through TestStack.
- Delete resources in dependency order:
- Delete function stacks first so their capacity-provider associations are detached.
- Delete the corresponding shared capacity provider stacks afterward.
- Revisit and retry stacks in
DELETE_FAILED.
- If deletion still fails after two attempts, file an issue or otherwise notify maintainers with the affected stack names and failure details.
The sweeper implementation can live in packages/testing and reuse the toolkit-lib machinery introduced by #5465.
Acknowledgment
Future readers
Please react with 👍 and your use case to help us understand customer demand.
Summary
Add a scheduled GitHub Actions workflow that removes leaked e2e infrastructure from the test account.
This is a follow-up to #5465, which introduces run-scoped shared Lambda Managed Instances (LMI) capacity provider stacks named
LmiShared-<runId>-<arch>. These stacks include EC2 fleets with a minimum capacity of 12 vCPUs and no scale-to-zero, a VPC, and a CloudWatch Logs interface endpoint. Unlike existing serverless e2e leftovers, leaked LMI infrastructure incurs hourly charges.This issue tracks only the scheduled sweeper. Complementary mitigations, including job
timeout-minutesand a destroy retry/second pass, are being handled during the review of #5465.Why is this needed?
A cancelled or timed-out LMI test cell can leave its per-suite function stack running, with the function attached to the shared capacity provider by ARN. Because this relationship crosses stacks and is invisible to CloudFormation, the
if: always()teardown job can encounterDELETE_FAILEDwhen deleting the provider stack. The current destroy script does not retry, and no later process revisits leaked resources in the account.Other leak paths include:
DISABLE_TEARDOWN.LmiShared-local-*stacks created from developer machines.A recurring cleanup mechanism is needed as a final safety net to limit cost and prevent abandoned e2e resources from accumulating.
This is a follow-up to #5465 and is related to #5092 and #5518.
Which area does this relate to?
Solution
Add a cron-based GitHub Actions workflow that runs nightly or every few hours and can also be invoked manually. It should assume the existing E2E OIDC role and require no standing AWS infrastructure of its own.
In
eu-west-1, the workflow should:LmiShared-*, includingLmiShared-local-*; orService: Powertools-for-AWS-e2e-tests, which all e2e stacks receive throughTestStack.DELETE_FAILED.The sweeper implementation can live in
packages/testingand reuse the toolkit-lib machinery introduced by #5465.Acknowledgment
Future readers
Please react with 👍 and your use case to help us understand customer demand.