[BUG] Harden OCI module publication, caching, parsing, and image trust

[doublewhy]

Description

The OCI module-registry and reference-backend paths have eight coupled reliability and trust defects:

1. published tar+gzip bundles inherit wall-clock gzip timestamps and host file metadata;
2. a relative or symlinked module entrypoint is not canonicalized against one stable bundle root;
3. extraction cache entries are neither bound to the layer digest nor protected by an inter-process lock, and extraction writes directly into the final cache directory;
4. publishing silently falls back to unsigned output when only one signing option is configured;
5. publishing writes into an existing OCI layout in place and retains stale inventory;
6. metadata fetches ignore the configured OCI timeout in favor of a separate constant;
7. malformed or non-UTF-8 registry JSON can escape as unstable decoder exceptions; and
8. the reference-backend image trust predicate accepts any string containing @sha256:, while runtime OSError failures are not classified by errno.

These defects can produce non-reproducible content digests, cache poisoning or partial-cache reuse, stale published blobs, unexpectedly unsigned modules, unbounded configuration drift, unstable public errors, and execution of malformed image references.

Requirements

• GOV-913 — Trust And Integrity Of Reusable Assets
• RUN-314 — reference backend

Steps to reproduce

1. Publish the same module twice at different wall-clock times; compare bundle and manifest digests.
2. Publish a relative path and a symlink to the same entrypoint; compare root_file and artifact identity.
3. Interrupt or concurrently run extraction for one manifest digest, or reuse a cache directory with a different layer digest.
4. pass signer_id without private_key_path, or the inverse.
5. Republish into a layout containing a stale blob or file.
6. replace the configured OCI limits timeout and inspect urlopen calls.
7. serve invalid UTF-8, malformed JSON, or a JSON scalar from tag, manifest, or config endpoints.
8. evaluate foo@sha256:<64 hex>:junk, sha256:<64 hex>, and placeholder-like malformed names with ImageTrustPolicy; inject PermissionError and non-availability OSError values into the OCI runner.

Expected behavior

• Bundle bytes and all content-addressed OCI descriptors are reproducible from identical canonical input bytes.
• Relative and symlink entrypoints resolve to one canonical source root and cannot escape it.
• A cache hit proves the expected layer digest, concurrent writers serialize, and readers see either a prior valid cache or a fully extracted replacement.
• Signing is explicitly both configured and valid, or explicitly absent.
• A successful publication exposes exactly the new layout inventory; failed publication preserves the prior layout.
• All OCI fetches use the validated configured timeout.
• All malformed registry JSON fails closed with stable, bounded SDLParseError messages.
• Only syntactically complete digest-pinned references or exact allowlist/default placeholder references pass image trust; runtime absence is distinguished from other command failures without leaking native error text.

Actual behavior

Current implementation violates each expectation above in the corresponding code path.

Gap Claim

The existing digest, signature, OCI-layout, extraction, and image-policy surfaces establish the right architectural boundaries, but several implementation details are not transactional, canonical, or grammar-complete. The fix should harden incumbents rather than add a second registry, cache, trust policy, exception hierarchy, or runtime abstraction.

Existing Surface Audit

• Module packaging and publication: raes.module_registry.publishing, models, signing, and digests.
• OCI resolution and extraction: raes.module_registry.resolution and constants.
• SDL error boundary: SDLParseError.
• Registry policy: RegistryTrustPolicy and raes-trust.yaml.
• OCI realization: raes_reference_backend.drivers.oci, ImageTrustPolicy, Diagnostic, and the injected subprocess runner.
• Public behavior: raes sdl publish/resolve/verify-imports, supply-chain trust specification, reference-backend guide, ADR-063, and ADR-071.
• Existing regression lanes: test_sdl_module_registry.py and test_reference_backend_oci_driver.py.

No second implementation stack is required.

Lineage / Precedent

• OCI import fetches are unbounded and allow memory/disk exhaustion #12 established bounded OCI reads and configurable OCI limits.
• OCI module resolver falls back to unsafe tar extraction on Python 3.11 #13 established safe, prevalidated tar extraction.
• OCI resolver does not verify config blob integrity and allows root_file tampering #14 bound config integrity and root_file into the signature payload.
• Trust & integrity of reusable assets (GOV-913) #115 / GOV-913 established reusable-asset identity, integrity, authenticity, bounded input, and redaction boundaries.
• Define the reference-backend contract and conformance evidence (RUN-314) #197 / RUN-314 established the reference OCI backend and fixed-argv, timeout, diagnostic, and trust boundaries.
• fix(reference-backend): anchor the OCI digest-pinned image-trust check #1084 contains the pending complete OCI image-reference grammar and targeted trust tests; integrate that logic while correcting OSError errno classification.

Literature / Practice

• OCI Image Layout defines oci-layout, index.json, and content-addressed blobs whose bytes match descriptor digests: https://github.com/opencontainers/image-spec/blob/main/image-layout.md
• OCI layer guidance says layers should be packed and unpacked reproducibly: https://github.com/opencontainers/image-spec/blob/main/config.md
• Python gzip documents mtime=0 for creation-time-independent output: https://docs.python.org/3/library/gzip.html
• Python tarfile exposes explicit member metadata and extraction filtering: https://docs.python.org/3/library/tarfile.html
• TUF describes consistent, signed, hash-bound repository metadata: https://theupdateframework.github.io/specification/latest/
• in-toto defines signed supply-chain metadata and verification boundaries: https://github.com/in-toto/specification/blob/master/in-toto-spec.md

Alternatives

1. Keep best-effort behavior and document it. Rejected because content digests and trust outcomes would remain nondeterministic or misleading.
2. Add a new external cache/database/registry client. Rejected because existing content-addressed paths and dependency-light architecture are sufficient.
3. Trust any substring containing a digest. Rejected because trust must bind a complete parsed reference, not an unanchored substring.
4. Delete the old layout/cache before rebuilding. Rejected because failure would destroy the last valid state.
5. Disable unsigned publication entirely. Rejected for compatibility; explicit neither-configured remains supported, while partial signing configuration fails closed.

Chosen Architecture

• Normalize archive order, POSIX paths, ownership, names, mode, mtime, gzip filename, gzip mtime, and compression level.
• Resolve the module entrypoint strictly once, derive one canonical bundle root, and enforce every imported path against it.
• Store a verified layer-digest marker inside the manifest-keyed cache; serialize rebuilds with a per-entry OS file lock; extract into a sibling staging directory; validate the root; then transactionally replace the cache.
• Treat signing options as a pair and translate missing/unreadable/invalid key material into stable SDLParseError failures.
• Construct the complete OCI layout in a sibling staging directory and transactionally publish it, with rollback preserving the old layout on failure and no stale inventory after success.
• Use the validated OCI limits timeout for every network request.
• Decode JSON through one bounded helper that catches UTF-8 and JSON shape failures without reflecting payloads or decoder internals.
• Integrate the full-match image-reference grammar from fix(reference-backend): anchor the OCI digest-pinned image-trust check #1084. Map availability errnos such as ENOENT, EACCES, EPERM, and ENOEXEC to runtime-missing; classify other OSError values as command-failed. Never expose native exception text.

Documentation Defense

Add a dated remediation/preflight note that maps each root cause to GOV-913 or RUN-314, the owning layer, negative tests, and stable public failure behavior. Update user-facing module publication and reference-backend documentation only where behavior is operator-visible. Do not create normative schema fields.

Verification Plan

• Regression tests for byte-identical bundles, normalized tar/gzip headers, relative/symlink canonicalization, and import escape rejection.
• Regression tests for partial signing configuration and invalid/unreadable keys.
• Regression tests for exact layout inventory, stale-file removal, and rollback on publication failure.
• Regression tests for configured fetch timeout and stable invalid UTF-8/malformed/non-object JSON errors.
• Regression tests for cache digest binding, partial-extraction cleanup, concurrent extraction, and cache replacement rollback.
• Port fix(reference-backend): anchor the OCI digest-pinned image-trust check #1084 image-reference positive/negative/placeholder tests; add availability and non-availability errno classification/redaction tests.
• Run focused pytest, Ruff, repository policy, requirement governance, and full verify_all checks.

Environment

• RAES SDL version: dev at 5c210d5
• Commit: 5c210d5
• Python version: project-supported matrix
• OS: cross-platform behavior; development reproduction on macOS

Architectural trust follow-up

Digest-pinned image admission establishes immutable content selection and integrity, but it does not by itself establish operator authorization or image authenticity. Preserve the focused ADR-063 grammar hardening from #1084, while explicitly deciding whether arbitrary plan-selected digests remain an accepted operator policy or must also satisfy registry allowlisting, signature verification, or another authenticity mechanism consistent with ADR-071. Any strengthened policy must remain operator-configured at the existing ImageTrustPolicy / backend registry seam and receive positive and negative tests without adding a parallel trust authority.

[doublewhy]

All eight defects are now resolved on dev:

1–7 landed in #1130 (deterministic bundles, canonicalized publishing root, digest-bound locked extraction cache, complete-signing-config requirement, atomic immutable layout publication, consistent OCI timeouts, normalized registry-metadata errors) — see that PR's verification section.

8 (reference-backend image trust + OSError classification) landed with the reference OCI driver hardening:

• raes_reference_backend/drivers/oci_image_trust.py anchors trust on a fullmatch of the OCI reference grammar terminated by a canonical 64-hex sha256: digest, so an unanchored @sha256: substring (evil/img@sha256:x/pull-me:latest, foo@sha256:short, wrong-case or wrong-length digests, trailing //: structure) is rejected rather than trusted. test_reference_backend_oci_driver.py pins exactly these attack strings, plus IPv6-authority and port-qualified accepts.
• Runtime OSError spawn failures are deliberately collapsed to one portable runtime-missing failure kind with the rationale documented at the catch site (drivers/oci.py): the native errno/strerror never crosses the portable driver boundary, which supersedes per-errno classification as the fix for the original 'unstable public errors' concern.

Verified today on dev (2d53aa34): the trust-predicate test module passes and the flagged code paths match the descriptions above.