test(release): refresh release-validation harness for v0.20.0 behavior changes#1758
Merged
Conversation
Bump pyproject.toml + uv.lock to 0.20.0 and roll the [Unreleased] CHANGELOG block into [0.20.0] - 2026-06-11. Lint mirror green locally (ruff check + format, pylint R0801, auth-signals). Post-merge: tag v0.20.0 to trigger the release workflow. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The release integration suite (which gates releases, not per-PR runs) surfaced 29 failures after v0.20.0 was tagged. Each was a stale test that had not been updated to match an intentional source behavior change merged this cycle. Source is correct throughout; only tests change here. Clusters and causing PRs: - A (#1735): download_github_file now calls _host._resolve_dep_auth_ctx; mock host helpers never stubbed it, forcing the wrong auth path. Added the stub to make_host/_make_host in 4 download test files. Also #1735 made generic raw-URL network errors raise RuntimeError instead of falling back to the Contents API -- rewrote 2 stale fallback tests to assert the raise. - B (#1742): apm compile -t copilot suppresses empty AGENTS.md shells when .github/instructions/*.md exists. Flipped 3 assertions to expect no AGENTS.md and assert the instruction files are present. - C (#1739): marketplace fetches moved to a shared requests.Session (_HTTP_SESSION.get); updated 6 tests to mock that seam. - D (#1734): optional registry env inputs are omitted without an override; updated 2 placeholder tests to assert the var is absent. - E (#1734): same omit-optional change; assert var not in result. - F (#1720): tar symlink rejection wording changed; updated the regex. Verified locally: 738 passed, 16 skipped (token-gated e2e). The two runnable cluster-B e2e tests (mixed_deps, guardrailing) pass against the v0.20.0 build; ado_e2e is identical logic, validated in CI. Full lint mirror green. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…r changes The release-validation shell harness carries its own copies of behavior assertions that duplicate the integration suite. Two of them went stale this cycle from the same PRs that broke the integration tests (#1757): - GH-AW compat (#1720): `apm pack --archive` now emits .zip by default; the archive check grepped only `build/*.tar.gz`. Accept either extension, testing each glob independently (a single `ls a b` exits non-zero when either pattern is unmatched, even if the other matches). - Hero scenario 2 / AGENTS.md (#1742): copilot `apm compile` omits the empty AGENTS.md shell when installed instructions already live under `.github/instructions/`. The check insisted AGENTS.md exist; now accept AGENTS.md OR a populated `.github/instructions/`, mirroring the merged pytest fix in test_guardrailing_hero_e2e.py. Same fixes applied to the Windows .ps1 (AGENTS.md only; it has no archive check). Predicates validated locally against apm v0.20.0. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
Pull request overview
Updates the v0.20.0 release/test surfaces to match recent intentional behavior changes (archive format defaulting to .zip, and Copilot-target compilation omitting empty AGENTS.md shells when .github/instructions/ is present), while also performing standard release metadata bumps.
Changes:
- Refresh release-validation harness assertions (bash + PowerShell) to accept either
AGENTS.mdor.github/instructions/*.md, and to accept.zipor.tar.gzpack artifacts. - Update multiple integration tests to reflect v0.20.0 behavior (optional env var omission, shared marketplace HTTP session seam, generic-host download error semantics, and AGENTS.md suppression).
- Bump version to
0.20.0and roll CHANGELOG entries into the0.20.0section.
Show a summary per file
| File | Description |
|---|---|
scripts/test-release-validation.sh |
Updates hero-scenario guardrail surface check; archive glob now accepts .zip or .tar.gz. |
scripts/windows/test-release-validation.ps1 |
Mirrors guardrail surface check logic for Windows harness. |
tests/integration/test_guardrailing_hero_e2e.py |
Updates assertions/logging to expect Copilot guardrails under .github/instructions/ rather than AGENTS.md. |
tests/integration/test_mixed_deps.py |
Updates mixed-deps compile expectation to omit AGENTS.md for Copilot instructions. |
tests/integration/test_ado_e2e.py |
Updates ADO compile expectation to validate .github/instructions/*.md and absence of AGENTS.md. |
tests/integration/test_download_strategies_selection.py |
Stubs new auth-context seam and updates generic-host raw failure expectation to raise a network error. |
tests/integration/test_download_strategies_phase3w5.py |
Same as above for the phase3w5 coverage suite. |
tests/integration/test_download_copilot_phase3.py |
Stubs auth seam; updates env-var placeholder behavior to omit optional vars without observed values. |
tests/integration/test_download_copilot_end_to_end.py |
Same as above for the end-to-end Copilot download test. |
tests/integration/test_wave7_policy_registry_coverage.py |
Updates optional env var behavior and marketplace HTTP mocking seam to _HTTP_SESSION.get. |
tests/integration/test_wave4_pure_logic_coverage.py |
Updates symlink tarball error-match string to the new wording. |
pyproject.toml |
Bumps package version to 0.20.0. |
uv.lock |
Updates locked editable package version to 0.20.0. |
CHANGELOG.md |
Adds 0.20.0 release heading and adjusts entry formatting to include the PR reference. |
Copilot's findings
- Files reviewed: 13/14 changed files
- Comments generated: 3
Comment on lines
5
to
8
| [project] | ||
| name = "apm-cli" | ||
| version = "0.19.0" | ||
| version = "0.20.0" | ||
| description = "MCP configuration tool" |
| @@ -124,7 +124,7 @@ class TestMixedDependencyCompile: | |||
| """Test compiling projects with mixed dependencies.""" | |||
|
|
|||
| def test_compile_with_mixed_deps_generates_agents_md(self, temp_project, apm_command): | |||
| @@ -156,7 +156,7 @@ class TestADOCompile: | |||
| ADO_TEST_REPO = "dev.azure.com/dmeppiel-org/market-js-app/_git/compliance-rules" | |||
|
|
|||
| def test_compile_generates_agents_md(self, tmp_path): | |||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
TL;DR
Fixes two stale assertions in the release-validation shell harness (
scripts/test-release-validation.sh+ Windows.ps1) that are blocking thev0.20.0release. Same root PRs as the #1757 integration fixes; these just live in a separate, shell-based test surface that per-PR CI never runs.Why the v0.20.0 release is still red
The harness keeps its own copies of behavior assertions that duplicate the integration suite, testing the shipped binary in isolation. Two went stale this cycle from intentional behavior changes:
build/*.tar.gzapm pack --archivenow emits.zipby defaultAGENTS.mdexistsapm compileomits the empty AGENTS.md shell when instructions already live under.github/instructions/This is why release run
27375730381showed integration passing on all platforms (the #1757 fix worked) but Release Validation (Linux) + Build & Validate (macOS) failing at "4/6 tests passed".What changed
scripts/test-release-validation.sh(Linux + macOS):build/*.zipORbuild/*.tar.gz. Tests each glob independently because a singlels a bexits non-zero when either pattern is unmatched (caught during local validation).AGENTS.mdexists, ELSE success if.github/instructions/*.mdexist, ELSE error. Mirrors the merged pytest fix intest_guardrailing_hero_e2e.py.scripts/windows/test-release-validation.ps1: same AGENTS.md logic in PowerShell (it has no archive check).Validation
apm v0.20.0: produced.zipmatches the OR-glob; instructions/AGENTS.md branches both pass..github/instructions/shape and passes.scripts/is not ruff-targeted, but the chain confirms no src drift.After merge
The
v0.20.0tag must be re-pointed to this commit's successor onmainand re-pushed to re-trigger the release pipeline. I will confirm with the operator before re-tagging rather than auto-retag.