CI: classify artifact capacity exhaustion separately from transient upload transport

[MrFr3di]

Problem

MuxTV's accepted evidence uploader (#141 / #176) correctly retries transient publication failures such as ECONNRESET and keeps evidence publication mandatory. The current repository failure mode is different and deterministic:

Failed to CreateArtifact: Artifact storage quota has been hit. Unable to upload any new artifacts. Usage is recalculated every 6-12 hours.

During the short 2026-08-25 self-hosted window this caused workflows with successful substantive work to finish red, including:

• docs: synchronize accepted stabilization truth #206 exact-head Run MuxTV validation = SUCCESS;
• chore(stack): stage Android stack refresh for runner validation #190 exact-head Full validation = SUCCESS;
• chore(stack): stage Android stack refresh for runner validation #190 exact API26/API36 product matrix = SUCCESS;
• artifact publication then failed because capacity was exhausted.

This must remain fail-closed for acceptance evidence, but must not be confused with a product/test failure or treated as a transient transport reset.

Current repository facts

The shared action .github/actions/upload-evidence-with-retry/action.yml already uses retention-days: 7 by default and performs one bounded retry plus REST reconciliation. This issue must not be framed as fixing a 90-day default retention problem.

#141 remains authoritative for transient transport/reconciliation. Capacity exhaustion is a separate state.

Official constraints

GitHub documents that:

• artifact storage is quota/billing capacity distinct from Actions cache storage and is shared with GitHub Packages allowance;
• deleting artifacts can reclaim current storage, but deletion is irreversible;
• storage usage/billing views can lag after deletion (GitHub documents a 6–12 hour update window);
• per-artifact retention is supported;
• artifact deletion requires Actions write permission.

References:

• https://docs.github.com/en/billing/concepts/product-billing/github-actions
• https://docs.github.com/en/actions/how-tos/manage-workflow-runs/remove-workflow-artifacts
• https://docs.github.com/en/rest/actions/artifacts

Required design

1. Preserve fail-closed acceptance semantics

If mandatory evidence cannot be published because capacity is exhausted, the workflow must still fail. Do not add continue-on-error as the final policy and do not claim acceptance GREEN.

2. Separate workload verdict from publication verdict

Emit a stable summary/diagnostic state that distinguishes at least:

• substantive workload/test failure;
• transient evidence transport failure eligible for bounded retry/reconciliation (CI: harden self-hosted evidence artifact upload against transient transport resets #141);
• artifact capacity exhaustion;
• evidence accepted/reconciled.

A red workflow caused by capacity must say explicitly that the substantive workload may have succeeded but acceptance evidence was not published.

3. Do not blindly retry deterministic capacity failures

If quota exhaustion can be classified reliably from supported action/API signals, stop before the second upload attempt and fail with the capacity-specific explanation. If actions/upload-artifact does not expose a stable machine-readable reason, do not parse fragile log prose in production merely to save one retry; first introduce a reliable preflight/API boundary or document that limitation.

4. Auditable cleanup only

Do not bulk-delete repository artifacts without an inventory and allowlist. Deletion is irreversible.

Before any cleanup automation, classify artifacts by provenance and value, for example:

• accepted main/manual acceptance evidence — longest retained class;
• active PR/device investigation evidence — bounded medium retention;
• superseded diagnostic/PR evidence — shortest retention / cleanup candidate.

Any cleanup implementation must be dry-run/read-only by default, report artifact id/name/run/created/expires/size when the API provides it, and require an explicit policy for destructive deletion.

5. Retention policy

Evaluate whether high-volume ephemeral PR/device artifacts should use a shorter explicit retention than the current shared default 7 days while durable acceptance evidence keeps a longer bounded retention. Do not apply a blanket reduction until actual artifact inventory shows it is justified.

Acceptance

• capacity exhaustion is distinguishable from product/test failure in job summary/logs;
• transient CI: harden self-hosted evidence artifact upload against transient transport resets #141 behavior remains intact;
• deterministic capacity failure is not blindly retried where a stable classifier exists;
• mandatory evidence remains fail-closed;
• no artifact is deleted without an auditable inventory/policy;
• no PAT/admin secret is added solely to make ordinary workflows look green;
• storage remediation does not create or modify Android AVD identities.

Ordering

This is CI signal/reliability work and may proceed runner-independently where testable. It must not modify frozen U0/#189 product source or reorder U0 -> U1 -> M0 -> #190 isolated landing.

[MrFr3di]

2026-08-26 inventory sampling — use class-based retention, not blanket cleanup

The connected GitHub surface does not expose repository-wide artifact listing/deletion, so no destructive cleanup was attempted.

Two concrete artifact samples establish a useful scale contrast:

• Focused Android TV device evidence, run 32664073541: artifact 9499653767, 13,919,066 bytes (~13.9 MB), created 2026-08-23, expires 2026-08-30 -> current 7-day heavy-device class.
• Accepted-main focused M3U evidence, run 31630259843: artifact 9154903369, 59,555 bytes (~60 KB), created 2026-08-12, expires 2026-09-11 -> legacy 30-day retention but negligible size compared with device evidence.

This is not a repository-wide inventory and must not be extrapolated into a total storage figure. It is enough to reject a blanket retention reduction: heavy device/validation artifacts and tiny textual evidence have materially different storage economics.

Recommended next evidence step before any delete/retention implementation:

1. obtain a complete artifact inventory (id, name, size, run, created_at, expires_at, expired) plus shared Packages usage where relevant;
2. group by workflow/evidence class and compute byte-days, not just artifact count;
3. preserve accepted/manual release evidence longest, keep active investigation evidence bounded, shorten only demonstrated high-volume ephemeral classes;
4. delete only an explicit superseded allowlist and remember GitHub documents deletion as irreversible.

Current shared uploader default remains 7 days. No retention or deletion code is changed by this comment.

[MrFr3di]

Architecture checkpoint after inspecting the pinned/current actions/upload-artifact@v7 contract and implementation.

Supported signal surface

upload-artifact@v7 exposes only:

• artifact-id;
• artifact-url;
• artifact-digest.

All are empty for any upload failure. The action delegates to @actions/artifact and lets upload exceptions escape; it does not expose a stable machine-readable failure kind such as quota-exhausted, HTTP status, retryable/non-retryable class, or capacity code.

Therefore the repository must not classify storage exhaustion by parsing the current prose Artifact storage quota has been hit.... That string is not an API contract and would make CI behavior depend on implementation text inside a third-party action.

Resulting design decision

Split #209 into two levels:

1. Implementable now with supported signals

   • keep publication fail-closed;
   • explicitly report accepted vs unaccepted-after-bounded-reconciliation;
   • include attempt outcomes/reconciliation reasons in the summary without secrets;
   • state that substantive workload verdict is separate from publication verdict;
   • preserve CI: harden self-hosted evidence artifact upload against transient transport resets #141 bounded retry because a failed action with empty outputs remains ambiguous.

2. Capacity-specific automatic classification

   • do not implement until there is a supported machine-readable signal/API available to ordinary repository CI credentials;
   • do not add PAT/admin secrets solely for this;
   • do not call undocumented Actions runtime endpoints;
   • do not parse vendor log prose.

This means the current deterministic capacity diagnosis remains an evidence-based human/owner classification from exact job logs, while repository automation should expose the narrower truth it can actually prove.

Retention finding remains unchanged

Measured samples show very different evidence sizes (heavy device evidence ~13.9 MB vs small accepted-main M3U evidence ~60 KB). Retention/cleanup should therefore remain class-based and inventory-first, not a blanket reduction.

Next implementation step when a PowerShell execution gate is available: RED contract for the new publication-state summary/output, then minimal change to the shared composite action. No Android runner/device is required for that test itself.