feat(garm): add idempotent resource cleanup

[yanksyoon]

Summary

• Add idempotent GARM scaleset/runner cleanup orchestration.
• Disable scalesets before draining runners.
• Poll asynchronous runner deletion and delete scalesets only after they are empty.
• Skip runners already pending deletion and preserve safe default force/bypass behavior.

Test plan

• pytest -q tests/unit/test_resource_cleanup.py tests/unit/test_garm_api.py

Stacked on #310 as part of ISD-6082. The charm remove-hook integration follows in the next PR.

Refs: ISD-6082

[Copilot]

Pull request overview

This PR adds an idempotent, retryable cleanup orchestrator for GARM scalesets/runners, intended to safely drain runners (handling asynchronous deletion) and only delete scalesets once they’re observed empty.

Changes:

• Introduces GarmResourceCleanup to disable scalesets, request eligible runner deletions, poll until empty, then delete scalesets.
• Adds unit tests covering the “drain then delete” flow and the “skip already pending deletion” flow.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

File	Description
charms/garm/src/resource_cleanup.py	Adds the cleanup orchestration logic, including deadline/poll loop and per-scaleset draining behavior.
charms/garm/tests/unit/test_resource_cleanup.py	Adds unit tests validating the cleanup orchestration ordering and idempotent behavior around pending deletions.

Suppressed comments (1)

charms/garm/tests/unit/test_resource_cleanup.py:65

• Unit tests in this charm consistently use AAA-style docstrings (arrange/act/assert), e.g. charms/garm/tests/unit/test_charm_state.py:93-97 and charms/garm/tests/unit/test_scaleset_reconciler.py:174-178. This new test uses a single-line docstring, which diverges from that convention.

    """Already pending runner deletion is observed until the row disappears."""

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.