design(tooling): define developer package and artifact management architecture

[Brad-Edwards]

Problem

OpenRAE currently acquires and publishes several classes of development and CI/CD artifacts through separately implemented paths: Python dependencies, repository tool binaries, GitHub Actions, proof tooling, OCI test inputs, build outputs, SBOMs, and release packages. Issue #1137 exposes the availability symptom, but adding a repository-owned HTTP transport would solve the wrong layer and create an unsafe maintenance boundary.

The repository needs one deliberate package/artifact-management architecture before it adds another downloader, cache, proxy, setup action, mirror, or publication path. Artifactory-style repositories are one solution class, not a preselected product. Other candidates include repository managers and pull-through caches, declarative CLI managers, OCI-backed distribution, maintained setup actions, native package managers, and combinations of those mechanisms.

Design scope

The design must inventory and classify every OpenRAE developer and CI/CD acquisition/publication surface, including at least:

• frozen Python dependencies and build environments;
• Conftest, Vale, Gitleaks, OSV Scanner, Isabelle, and other repository/proof tools;
• third-party GitHub Actions and reusable workflows;
• OCI images used as required test/release inputs;
• wheels, source distributions, SBOMs, GitHub Releases, and PyPI publication;
• caches, generated reports, and ephemeral workflow artifacts, explicitly distinguishing those that do and do not belong in the managed package boundary.

Runtime-domain package declarations, reusable scenario/module semantics, and backend workload package modeling remain separate unless the design demonstrates a shared infrastructure boundary without conflating their authorities.

Operating contexts and questions

The design must support or explicitly disposition:

• public contributors and GitHub-hosted CI;
• local development and test environments;
• self-hosted, high-load, concurrent, and multi-process runners;
• offline and air-gapped operation, including import/export and pre-seeding;
• authenticated enterprise mirrors without exposing credentials to untrusted pull requests;
• upstream outages, rate limits, repository relocation, artifact removal, and cache loss;
• Linux/macOS and every architecture the repository claims for a given tool;
• immutable version/digest selection, checksums, signatures, attestations, SBOMs, and provenance;
• bootstrap trust for the package manager or installer itself;
• promotion from upstream intake through verification to an immutable internal/public distribution surface;
• retention, revocation/quarantine, garbage collection, disaster recovery, observability, and ownership;
• an explicit no-custom-HTTP-stack rule: network retry, proxy, redirect, TLS, framing, and timeout behavior must come from maintained commodity tooling or infrastructure, not repository-authored transport code.

Required outputs

1. A repository-wide acquisition and publication inventory with owners, consumers, trust roots, availability requirements, and current duplication.
2. A decision matrix comparing solution classes rather than assuming a product. At minimum compare:
   • repository/proxy managers such as Artifactory-, Nexus-, or Cloudsmith-class systems;
   • declarative developer CLI managers such as Aqua- or Hermit-class systems;
   • OCI registry/artifact distribution;
   • upstream-maintained setup actions/workflows and native package managers;
   • direct pinned acquisition plus hosted/local caches where it remains justified.
3. A selected architecture that separates:
   • the repository manifest/lock and integrity authority;
   • upstream intake and verification;
   • storage/mirroring/promotion;
   • client installation and local cache behavior;
   • CI orchestration and release publication.
4. ADRs and supporting architecture/design documents describing the decision, nonclaims, security/availability model, failure behavior, migration, and operations.
5. One implementation issue per independently reviewable change, all assigned to this milestone.
6. A complete GitHub issue-dependency graph. Every applicable open issue and newly created implementation issue must use GitHub's native blocking relationships; prose-only depends on references are insufficient.
7. A migration/disposition for the existing acquisition implementations and PR fix: retry transient pinned tool downloads #1140, including removal or rejection of repository-owned HTTP transport logic.

Acceptance criteria

• Every repository development/CI/release acquisition and publication surface is inventoried and classified.
• Requirements cover public, local, self-hosted, high-load/concurrent, offline, and air-gapped operation.
• At least the solution classes named above are evaluated against the same security, availability, portability, operational, and maintenance criteria.
• The selected architecture identifies authoritative manifests/locks, trust roots, cache/repository roles, promotion flow, fallback policy, and bootstrap mechanism.
• Artifact repositories and caches are treated as availability/distribution controls; repository-reviewed digests, signatures, or attestations remain the integrity authority unless an ADR explicitly replaces that authority.
• No custom HTTP client or transport stack is introduced or retained for package/artifact acquisition.
• ADRs and architecture documents are accepted before implementation begins.
• Existing applicable issues are moved to this milestone and blocked on this design issue where appropriate.
• All implementation issues are created in this milestone with native GitHub blocking relationships that encode the complete execution order.
• Make pinned CI tool downloads resilient to transient upstream failures #1137 receives an explicit disposition under the selected architecture rather than a larger downloader implementation.
• Release publication issues cannot proceed before their exact-SHA, required-test-input, and artifact-admission prerequisites.
• The final design includes migration, rollback, operations, and acceptance tests for every supported operating context.

Initially related work

• Make pinned CI tool downloads resilient to transient upstream failures #1137 — pinned repository-tool acquisition reliability
• ci(release): require a real digest-pinned container integration lane #1110 — digest-pinned OCI release-test input
• ci(release): bind publication to the exact verified release SHA #1125 — exact-SHA build and publication admission
• Publish aces-sdl to PyPI and auto-release on merge to main (conventional-commit model, per aces-scenario-packs ADR 0006) #684 — PyPI/GitHub release publication
• Add OpenSSF Scorecard workflow and README badge #839 — third-party CI action introduction
• PR fix: retry transient pinned tool downloads #1140 — proposed custom release-download transport; deferred for disposition by this design