Optimized whole-app test suite: coverage map, parallel shards, and time budget

[NtrpyDev]

Blocked-by: #47 #48 #50

Goal

Build one optimized test system that proves every observable 1.0.0 product claim and finishes quickly enough to run throughout development.

Required investigation and implementation

1. Inventory every shipped surface and behavior: daemon CLI, private Unix-socket API, native GTK4 client, setup, Steam and GSI discovery, import and analysis, storage and recovery, bundled recorder build and capture supervision, Manual Flag, clip review/trim/export, desktop and user service integration, static install, AUR, COPR, release artifacts, GPL notices, privacy, local-only operation, upgrades, and uninstalls.
2. Publish a machine-checkable coverage map that links each behavior and important failure mode to at least one test. The suite must fail when a required behavior has no verifier.
3. Separate the suite into the smallest useful hermetic shards. Run independent shards concurrently, serialize only proven shared-state boundaries, reuse compiled artifacts, and avoid compiling the same Rust graph once per packaging target.
4. Add deterministic fixtures and fakes for Steam metadata, CS2 demos and GSI events, FFmpeg and FFprobe, capture, portals, filesystem failures, process failures, and time. Do not depend on Noah's home directory, live Steam state, desktop session, network, or credentials.
5. Cover happy paths, invalid input, authorization, deduplication, concurrency, restart and recovery, interrupted writes, partial failures, privacy and secret handling, loopback binding, remote endpoint rejection, package install and upgrade, package removal with preserved user data, and release/source reproducibility.
6. Keep true headless native GTK journeys, accessibility checks, clean package environment, and release tests, but move expensive checks out of duplicated paths and feed them prebuilt inputs where that does not weaken what they prove.
7. Measure cold and warm baseline wall time before optimization, record per-shard timing, set an explicit local and CI time budget, and fail on material regression. Report critical path and slowest tests.
8. Preserve complete failure output from parallel shards. Do not use retries, ignored failures, broad exclusions, or reduced assertions to manufacture speed.

Acceptance criteria

• One documented command runs the complete 1.0.0 verifier, and one fast command runs the developer feedback subset.
• The complete verifier's coverage map has no uncovered required behavior or failure-mode row.
• Independent test shards execute concurrently with bounded resource use and deterministic ports, homes, and temporary directories.
• Rust compilation and release binaries are reused across compatible shards; package environments consume exact prebuilt source inputs.
• Repeated clean runs prove the suite is stable, and cold/warm timing evidence demonstrates the optimized critical path.
• GitHub Actions runs the same shard definitions as local execution and cannot report success while any shard is missing, cancelled, timed out, or failed. The complete graph must finish comfortably inside the job timeout on the smallest supported runner.
• Existing assertions and supported-platform checks remain intact unless replaced by equal or stronger proof.

Resolution notes required

Record the final coverage inventory, shard graph, baseline and optimized timings, concurrency limit, deliberately serialized boundaries, and any behavior deferred beyond 1.0.0.

Reopened after the v1 architecture pivot

The prior resolution covered the removed browser dashboard and external-recorder model. It is historical timing evidence only. The ticket is reopened because v1 now ships a GTK4 client, private Unix socket, and pinned recorder source. The cancelled 30-minute GitHub run at https://github.com/NtrpyDev/openfrag/actions/runs/29434019020 also proves the current CI budget is not met.

The revised suite must add native widget and presenter coverage, Unix-socket authorization and lifecycle tests, accessibility-tree assertions, headless GTK journeys, bundled-recorder compilation and package-content checks, portal and codec fakes, installed three-process lifecycle tests, and stale-web-surface rejection. Remove Chromium and Node preparation once no browser surface remains. Rebaseline cold and warm timing after the final architecture lands, then prove repeated complete runs locally and in GitHub Actions.

[NtrpyDev]

Resolved on implementation/v1-integrated through dd98424.

Coverage inventory

• 27 machine-checked requirement rows cover all 17 required surfaces: daemon CLI, loopback API, dashboard, setup, Steam and GSI, import and analysis, storage and recovery, capture supervision, Manual Flag, clip workflow, desktop service, static install, AUR, COPR, release artifacts, licensing and privacy, and upgrade and uninstall behavior.
• Validation rejects a missing required surface, behavior, failure mode, verifier, evidence file, complete-profile verifier, duplicate ID, unknown resource, or invalid local or CI budget.

Shard graph

• Serialized preparation: suite contract, first-party format and metadata checks, complete-profile clippy, one shared Rust test graph, immutable production and acceptance binaries, and one deterministic vendored source archive.
• Concurrent verification: Rust domain, runtime, and daemon; parser contracts; static packaging; release reproducibility; production headless; installed MVP; real Chromium dashboard; clean AUR; clean Fedora 43; and clean Fedora 44.
• General concurrency is capped at four jobs. Native package work has a separate two-job semaphore. Every shard gets an isolated HOME, temporary tree, XDG tree, deterministic loopback port, and retained log.
• Compatible shards reuse one Cargo target and immutable prepared binaries. Package environments build the exact prepared source archive.

Timing evidence on PC1

• Previous serial baseline: 180.53 seconds cold and 22.46 seconds warm. A duplicated cold attempt also exhausted the 16 GB temporary filesystem after 159.57 seconds.
• Optimized fast profile: 45.32 seconds cold and 3.58 seconds warm, within the 120-second local budget.
• Two consecutive complete runs passed in 787.16 and 797.85 seconds, within the 1,200-second local budget. The first run's slowest phases were Fedora 44 at 502.16 seconds, Fedora 43 at 499.79 seconds, AUR at 269.91 seconds, shared build preparation at 11.28 seconds, and release reproducibility at 11.10 seconds.
• The observed complete critical path is serialized preparation followed by the bounded native-package queue.

GitHub Actions installs the pinned browser harness and invokes the same complete manifest. The runner retains full output, does not fail fast across independent shards, and fails for failed, missing, timed-out, cancelled, or over-budget work.

No v1 software behavior is deferred. Physical gates requiring a real CS2 session, approved desktop portals, capture and audio selection, playback, or human observation remain in #45 because unattended fixtures cannot honestly qualify those host interactions.