ADR-0002: Generate certification documents from CI output and withdraw the current false certifications

[nicholas-ruest]

Implements docs/adr/ADR-0002-certification-document-integrity.md.

What changed

Fixed the three failing tests, in the logic, not the assertions.

• ResolutionConfig used #[derive(Default)], which ignores the #[serde(default = "...")] attributes on its own fields and yields false/0. So ResolutionConfig::default() disabled parallel-group generation, critical-path computation and cycle detection, and set max_depth to 0 — and resolve() returned empty parallel_groups for any graph it had just resolved successfully. Replaced with a hand-written impl Default matching the serde defaults, as the sibling DependencyResolverConfig and agentics_contracts::DependencyResolutionConfig already do.
• StateMachineAgent::transition reported every rejected transition with an existing source and target as Blocked. TransitionStatus documents Blocked as "transition was blocked by a rule" — it is for an edge the machine permits but a guard rejects. An edge the machine does not declare is Invalid. Classification now asks the state machine directly.
• A self-transition (running → running) was rejected because the machine declares no self-loop edge. Requesting the state an entity is already in changes nothing and cannot fail; validate_transition now treats current == target as valid, which lets transition() reach its existing NoChange branch.

No assertion was weakened. Each fix is its own commit.

Built the generator. scripts/generate-test-certification.py reads a libtest-JSON event stream, aggregates per crate, extracts each failure's assertion site, and emits docs/TEST_CERTIFICATION.md with a provenance header (commit, CI run ID and URL, exact command, toolchain, UTC timestamp). One verdict line: PASS only when nothing failed, else FAIL (N failing). It never writes "production ready", "certified", or a grade. 19 unit tests over two fixtures recorded from real nextest runs — one all-pass (198/198), one taken from the tree before the fixes (195/198) — assert it reports FAIL (3 failing) and names every failing test with its site.

Built the reintroduction guard. scripts/check-certification-integrity.py fails on any file under docs/ using readiness vocabulary without a provenance header. ADRs and runbooks are exempt (they assert intent, not measurement).

Withdrew and retired the false certifications, then corrected the same claims in three more documents (below).

Real test output

Command: cargo test -p llm-orchestrator-core --lib

	Before	After
passed	195	198
failed	3	0
total	198	198

test result: ok. 198 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.32s

Workspace-wide, cargo nextest run --all --lib --bins --no-fail-fast:

Summary [26.290s] 435 tests run: 432 passed, 3 failed, 2 skipped

llm-orchestrator-core is 198/198. The generated docs/TEST_CERTIFICATION.md in this PR is that run.

Generator tests: Ran 19 tests ... OK. Guard: No unprovenanced readiness claims under docs.

Three things this uncovered that the ADR did not anticipate

1. cargo test --all does not compile, and has not since January. crates/agentics-contracts/tests/integration_tests.rs fails with 64 errors against an API that moved out from under it (DecisionOutputs, WorkflowExecutionConfig, RetryConfiguration, DependencyResolutionOutputs no longer exist; ConstraintsApplied renamed its fields). Last touched by 63ae636 (2026-01-20). This is why the CI Test job has been failing on main — e.g. run 22251502561 shows Test (beta) failed and Test (stable) cancelled.

ADR-0002 verification criterion 1 requires cargo test --all to be green. It is not, and this PR does not make it so. Repairing that test file is a separate body of work requiring judgement about what it should assert, so I stopped rather than guess. The certification run is scoped to --lib --bins and the provenance header records that exact command, so the document does not overstate what was measured.

2. Three more real failures outside llm-orchestrator-core. ADR-0002 only ever ran -p llm-orchestrator-core --lib, so these had never been measured:

Test	Site
llm-orchestrator-providers::cohere_embeddings::tests::test_multilingual_model	crates/llm-orchestrator-providers/src/cohere_embeddings.rs:601
llm-orchestrator-providers::cohere_embeddings::tests::test_search_query_input_type	crates/llm-orchestrator-providers/src/cohere_embeddings.rs:433
llm-orchestrator-secrets::cache::tests::test_cache_expiration	crates/llm-orchestrator-secrets/src/cache.rs:331

Both cohere tests abort the process (SIGABRT, "panic in a destructor during cleanup") under mockito; the cache test asserts misses == 2 with a 100 ms TTL and a 150 ms sleep, and is timing-sensitive. All three reproduce under plain cargo test, so they are not a nextest artefact. Out of ADR-0002's scope, left unfixed, and now visible in the certification — which is the point of the mechanism.

3. The false figures were in five documents, not two. The guard found 234/234 in IMMEDIATE_ACTIONS_COMPLETE.md, 243/243 in PHASE_3_IMPLEMENTATION_COMPLETE.md, and a certification level plus a project score in PHASE_4_COMPLETE.md. That is a third and fourth mutually inconsistent total for the same tree (204+, 243, 234), strengthening the ADR's argument. I withdrew their readiness and test-result claims in place and left the surrounding narrative alone — these are records of work done, not certifications. Ten further documents carrying ungrounded N/100 scores (pentest reports, phase summaries) are grandfathered in the guard as an explicit, commented debt list, so the guard lands as a ratchet rather than a rewrite of a dozen documents the ADR never examined.

⚠️ Left undone: the CI wiring (ADR step 4) — needs a human

The push was rejected: this token is a fine-grained PAT without workflow permission, so .github/workflows/ci.yml could not be modified:

! [remote rejected] ... (refusing to allow a Personal Access Token to create or
  update workflow `.github/workflows/ci.yml` without `workflow` scope)

The generator and guard are therefore committed but not enforced. Someone with workflow permission needs to apply the diff below. Until they do, criteria 3 and 4 of the ADR's Verification section are satisfied only by the generator's unit tests, not by CI.

Intended .github/workflows/ci.yml diff (tested locally end to end)

@@ jobs.test.steps @@
+      - name: Install cargo-nextest
+        uses: taiki-e/install-action@nextest
+
       - name: Build
         run: cargo build --all --verbose
 
-      - name: Run tests
-        run: cargo test --all --verbose
+      # Machine-readable output is the ground truth the certification document is generated
+      # from. `--no-fail-fast` so a failing crate does not hide the rest of the suite, and
+      # `|| true` so the step reaching the generator does not depend on the suite passing --
+      # the verdict comes from the generator's exit code, checked below.
+      #
+      # Scoped to --lib --bins because `agentics-contracts`'s integration_tests target has not
+      # compiled since 63ae636 (2026-01-20); including it aborts the build before a single test
+      # runs, which is what has kept this job red. The next step keeps that breakage visible
+      # rather than hiding it behind the narrower scope.
+      - name: Run tests
+        run: |
+          cargo nextest run --all --lib --bins --no-fail-fast --message-format libtest-json \
+            > nextest-events.json || true
+        env:
+          NEXTEST_EXPERIMENTAL_LIBTEST_JSON: 1
+
+      # Non-blocking so the certification check below is the job's authoritative verdict, but
+      # kept in the job so the breakage stays on the dashboard instead of disappearing.
+      # Remove continue-on-error once agentics-contracts::integration_tests compiles again.
+      - name: Build integration test targets (known broken -- see ADR-0002)
+        run: cargo test --all --no-run
+        continue-on-error: true
+
+      - name: Generate test certification
+        if: always()
+        run: |
+          python3 scripts/generate-test-certification.py \
+            --commit "${{ github.sha }}" \
+            --run-id "${{ github.run_id }}" \
+            --run-url "${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}" \
+            --command "cargo nextest run --all --lib --bins --no-fail-fast --message-format libtest-json" \
+            --toolchain "$(rustc --version)" \
+            --output docs/TEST_CERTIFICATION.md \
+            < nextest-events.json
+        continue-on-error: true
+
+      - name: Upload test certification
+        if: always()
+        uses: actions/upload-artifact@v4
+        with:
+          name: test-certification-${{ matrix.rust }}
+          path: docs/TEST_CERTIFICATION.md
+
+      # The committed document must agree with this run's measurements. Provenance rows differ
+      # every run by construction and are excluded; the numbers are not. A mismatch means the
+      # file was hand-edited or left stale.
+      - name: Verify committed certification matches this run
+        if: always()
+        run: |
+          git checkout -- docs/TEST_CERTIFICATION.md
+          python3 scripts/generate-test-certification.py \
+            --commit "${{ github.sha }}" \
+            --run-id "${{ github.run_id }}" \
+            --run-url "${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}" \
+            --command "cargo nextest run --all --lib --bins --no-fail-fast --message-format libtest-json" \
+            --toolchain "$(rustc --version)" \
+            --check docs/TEST_CERTIFICATION.md \
+            < nextest-events.json
 
       - name: Run doc tests
+        if: always()
         run: cargo test --doc --verbose
 
+  certification-integrity:
+    name: Certification Integrity
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Test the certification generator
+        run: python3 -m unittest discover -s tests/certification -v
+
+      # Stops hand-authored readiness claims from reappearing in docs/.
+      - name: Check for unprovenanced readiness claims
+        run: python3 scripts/check-certification-integrity.py docs
+
   lint:

Deviations from the ADR, and why

--check instead of git diff --exit-code. Step 4 specifies git diff --exit-code docs/TEST_CERTIFICATION.md. That can never pass: the provenance header contains the commit SHA, CI run ID, run URL and generation timestamp, all of which differ on every run by construction, so the diff would always be non-empty and the signal would be pure noise. The generator instead grows a --check mode comparing only the measured sections (verdict, per-crate table, failure list). This preserves the property the ADR is after — a hand-edited number fails the build — and is covered by three unit tests, including one for a stale green document left in place after the suite regressed. Assertion sites are normalised to repo-relative paths for the same reason: an absolute path would embed the runner's home directory.

Documents stubbed, not deleted. Step 7 says delete. Three disaster-recovery documents link to PRODUCTION_READINESS_CERTIFICATION.md; deleting it breaks them. Both files are reduced to a short pointer at docs/TEST_CERTIFICATION.md and those inbound links now target the generated document. Content remains in git history and is not resurrected with new hand-written numbers.

The committed certification names a local run, not a CI run. It has to: CI cannot be wired from here. Its provenance says so explicitly rather than pretending otherwise. Once the CI diff above is applied, the first run will regenerate it; if test_cache_expiration (timing-sensitive) behaves differently on CI hardware, the --check step will correctly flag the mismatch and CI's own artefact should be committed in its place.

ADR Verification criteria

#	Criterion	Status
1	cargo test -p llm-orchestrator-core --lib → 198/0	✅
1	cargo test --all green	❌ pre-existing breakage, see above
2	TEST_CERTIFICATION.md totals equal a fresh run	✅ verified by --check round-trip
3	A broken test yields FAIL (N failing) naming it	✅ unit-tested; not yet in CI
4	Hand-editing a number fails the check	✅ unit-tested; not yet in CI
5	grep -rniE "certified production-ready|243/243|204/204|100% \(all passing\)|PLATINUM" docs/ clean outside the ADR	✅
6	No doc asserts a count or pass rate without provenance	⚠️ enforced, with 10 grandfathered files listed in the guard
7	ADR-0001 records the dependency	✅

The target/ directory was removed after building; the working tree is clean.