test: extend cross-collection maturity-conflict coverage#2219
test: extend cross-collection maturity-conflict coverage#2219PratikWayase wants to merge 2 commits into
Conversation
…Collections.Tests.ps1
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #2219 +/- ##
==========================================
+ Coverage 81.29% 81.38% +0.09%
==========================================
Files 129 119 -10
Lines 19068 19016 -52
Branches 12 0 -12
==========================================
- Hits 15501 15477 -24
+ Misses 3564 3539 -25
+ Partials 3 0 -3
Flags with carried forward coverage won't be shown. Click here to find out more.
🚀 New features to boost your workflow:
|
rezatnoMsirhC
left a comment
rezatnoMsirhC
left a comment
There was a problem hiding this comment.
Thanks for the contribution. The required automated checks are all unchecked in the PR description. Please run at minimum npm run lint:ps (the dead $effectiveMaturity assignment will fail it), check the corresponding boxes, and rebase onto main before requesting merge.
| } | ||
| } | ||
|
|
||
| if ($themedMatches.Count -gt 1) { |
There was a problem hiding this comment.
This new block is placed after the continue in Check 4 (~line 445), so items absent from the canonical collection never reach it. Real-world case: item A is in three themed collections with conflicting maturities and absent from hve-core-all. Check 4 fires and continues, making the cross-themed disagreement invisible. All five new tests put the item in canonical, so none exercise this gap.
Suggested fix: move this block above the Check 4 guard so it runs unconditionally.
| $crossCheckMaturity = [string]$manifest.maturity | ||
| } | ||
|
|
||
| $effectiveMaturity = Resolve-CollectionItemMaturity -Maturity $itemMaturity |
There was a problem hiding this comment.
$effectiveMaturity is assigned here but is no longer consumed. Its only use (Maturity = $effectiveMaturity) was replaced in the occurrence record below. PSUseDeclaredVarsMoreThanAssignments will fail npm run lint:ps.
Either remove this line, or thread the resolved 'stable' default back into $crossCheckMaturity as a final fallback when neither item nor collection maturity is set.
| Kind = $kind | ||
| Path = $itemPath | ||
| Maturity = $effectiveMaturity | ||
| Maturity = $crossCheckMaturity |
There was a problem hiding this comment.
Storing $crossCheckMaturity here instead of $effectiveMaturity silently changes behaviour for two downstream checks:
- Check 4: items in a collection with
maturity: removedat collection level now resolve to'removed', so they are excluded from$activeThemedMatchesand the "missing from canonical" error is suppressed. - Orphan check: those same items now hit the
$themedRemovedpath, skipping orphan detection.
If intentional, please add a comment and a test documenting this. If not, consider keeping $effectiveMaturity here and using $crossCheckMaturity only within the conflict-detection blocks.
| $kind = $item.kind | ||
| $absolutePath = Join-Path -Path $RepoRoot -ChildPath $itemPath | ||
| $itemMaturity = $null | ||
| $itemMaturity = $null |
There was a problem hiding this comment.
Nitpick: this line has 24 leading spaces; the adjacent $isExplicitItemMaturity = $false on the next line has 12. Restoring to 12 spaces keeps PSUseConsistentIndentation clean.
| } | ||
| } | ||
|
|
||
| Describe 'Invoke-CollectionValidation - cross-collection maturity conflicts' { |
There was a problem hiding this comment.
Missing -Tag 'Unit'. Per pester.instructions.md and the pattern set by the most-recently added Describe block in this file (Invoke-CollectionValidation - marker validation at line 1282), all blocks require -Tag 'Unit'. Without it, tag-filtered Pester runs will silently skip these five tests.
Describe 'Invoke-CollectionValidation - cross-collection maturity conflicts' -Tag 'Unit' {| $errorString | Should -Match 'deprecated' | ||
| } | ||
|
|
||
| It 'Excludes hve-core-all canonical collection from conflict detection (Regression Guard)' { |
There was a problem hiding this comment.
This test guards the $canonical.IsExplicitItemMaturity path, not the structural canonical-exclusion invariant described in the test name. hve-core-all is already excluded from $themedMatches by the $canonicalCollectionId filter. It can never appear there regardless of this guard. With only one themed collection, the new cross-themed check is not exercised at all.
Suggested addition: assert that no CrossCollectionMaturityConflict error lists hve-core-all as a participant collection ID. A scenario with two conflicting themed collections AND hve-core-all in canonical would fully exercise the structural invariant.
|
|
||
| $agentsDir = Join-Path $script:repoRoot '.github/agents/test' | ||
| New-Item -ItemType Directory -Path $agentsDir -Force | Out-Null | ||
| Set-Content -Path (Join-Path $agentsDir 'a.agent.md') -Value '---\ndescription: shared agent\n---' |
There was a problem hiding this comment.
Nitpick: single-quoted strings in PowerShell do not expand \n: the file will contain literal backslash-n characters, not valid YAML frontmatter. Currently harmless since the validator only checks path existence, but would cause spurious fixture failures if file content is ever inspected. This pattern exists throughout the existing test file, so tagging @microsoft/edge-ai-core-dev for input on whether to standardise on here-strings in new tests or keep the established convention.
4 participants
Description
Follow-up bundle from the
owasp-dockerremoval /maturity: removedrefactor. This PR extends the validation logic and test coverage to prevent silent maturity drift as more themed collections adopt thematurity: removedmarker at the collection YAML level.Test Coverage (
Validate-Collections.Tests.ps1):removedvsstable,removedvsdeprecated,deprecatedvsexperimental) with clear error message assertions.hve-core-allcollection is explicitly excluded from the new collection-level conflict detection.Implementation (
Validate-Collections.ps1):Related Issue(s)
Closes #1446
Type of Change
Select all that apply:
Code & Documentation:
Infrastructure & Configuration:
AI Artifacts:
prompt-builderagent and addressed all feedback.github/instructions/*.instructions.md).github/prompts/*.prompt.md).github/agents/*.agent.md).github/skills/*/SKILL.md).github/hooks/*/*.json)evals/)Other:
.ps1,.sh,.py)Testing
Ran the Pester test suite locally using
Invoke-Pester -Path "scripts/tests/collections/Validate-Collections.Tests.ps1". All 63 tests pass successfully, including:hve-core-allexclusion regression guard).Checklist
Required Checks
AI Artifact Contributions
/prompt-analyzeto review contributionprompt-builderreviewRequired Automated Checks
The following validation commands must pass before merging:
npm run lint:mdnpm run spell-checknpm run lint:frontmatternpm run validate:skillsnpm run lint:md-linksnpm run lint:psnpm run eval:lint:schemanpm run plugin:generatenpm run docs:testSecurity Considerations
Additional Notes
None.