Skip to content

Add pin-policy-bundle acceptance tests for TA task#3324

Open
joejstuart wants to merge 2 commits into
conforma:mainfrom
joejstuart:policy-digest-acceptance
Open

Add pin-policy-bundle acceptance tests for TA task#3324
joejstuart wants to merge 2 commits into
conforma:mainfrom
joejstuart:policy-digest-acceptance

Conversation

@joejstuart

Copy link
Copy Markdown
Contributor

Summary

  • Add pin-policy-bundle step log assertion to the existing "Golden container image with trusted artifacts" scenario (no-op path: policy uses git sources, OCI tag not found)
  • Add new "Pin policy bundle digest" scenario for verify-conforma-konflux-ta task (replacement path: OCI tag replaced with digest-pinned reference)
  • Add corresponding snapshot entries

These mirror the two pin-policy-bundle scenarios already covered by the verify-enterprise-contract task, closing a gap where the TA task variant had zero dedicated coverage for this step.

Test plan

  • Acceptance tests pass for ta_task_validate_image feature

🤖 Generated with Claude Code

@coderabbitai

coderabbitai Bot commented May 26, 2026

Copy link
Copy Markdown

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Enterprise

Run ID: 4fc34792-de54-4478-9e72-a966e5d9ac33

📥 Commits

Reviewing files that changed from the base of the PR and between ceabb47 and 5d85d39.

⛔ Files ignored due to path filters (1)
  • features/__snapshots__/ta_task_validate_image.snap is excluded by !**/*.snap
📒 Files selected for processing (1)
  • features/ta_task_validate_image.feature
🚧 Files skipped from review as they are similar to previous changes (1)
  • features/ta_task_validate_image.feature

📝 Walkthrough

Walkthrough

The feature tests add snapshot coverage for the pin-policy-bundle step and introduce a scenario validating explicit policy bundle digest handling with non-strict mode.

Changes

Policy bundle validation

Layer / File(s) Summary
Trusted-artifact snapshot assertion
features/ta_task_validate_image.feature
The existing trusted-artifact scenario now checks pin-policy-bundle logs against the snapshot.
Explicit digest scenario
features/ta_task_validate_image.feature
A new scenario runs with POLICY_BUNDLE_DIGEST and STRICT=false, then verifies task success and pin-policy-bundle and show-config snapshots.

Estimated code review effort: 2 (Simple) | ~10 minutes

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly summarizes the main change: adding pin-policy-bundle acceptance tests for the TA task.
Description check ✅ Passed The description accurately matches the changes by describing the new pin-policy-bundle assertions and scenario coverage.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands.

@codecov

codecov Bot commented May 26, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

Flag Coverage Δ
acceptance 54.26% <ø> (ø)
generative 16.80% <ø> (ø)
integration 27.97% <ø> (ø)
unit 71.74% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.
see 1 file with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@joejstuart

Copy link
Copy Markdown
Contributor Author

/retest

The verify-conforma-konflux-ta task had no test coverage for the
pin-policy-bundle step. Mirror the two scenarios already covered by the
verify-enterprise-contract task: the no-op path (policy uses git
sources) and the replacement path (policy uses the OCI tag reference).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@simonbaird simonbaird force-pushed the policy-digest-acceptance branch from 36c805b to ceabb47 Compare July 14, 2026 18:24
@simonbaird

Copy link
Copy Markdown
Member

Gave it a rebase and a conflict resolve.

@fullsend-ai-review

fullsend-ai-review Bot commented Jul 14, 2026

Copy link
Copy Markdown

🤖 Finished Review · ✅ Success · Started 6:25 PM UTC · Completed 6:29 PM UTC
Commit: 87c4a29 · View workflow run →

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@features/ta_task_validate_image.feature`:
- Line 131: Update the POLICY_BUNDLE_DIGEST scenario value in the relevant
feature example to a valid digest different from the task default, then
regenerate or edit the expected snapshots to reflect that supplied digest and
verify the scenario confirms explicit parameter propagation.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Enterprise

Run ID: 2f2c6e31-abf4-4b00-b5d8-d6a9892ca6a0

📥 Commits

Reviewing files that changed from the base of the PR and between 9587783 and ceabb47.

⛔ Files ignored due to path filters (1)
  • features/__snapshots__/ta_task_validate_image.snap is excluded by !**/*.snap
📒 Files selected for processing (1)
  • features/ta_task_validate_image.feature

Comment thread features/ta_task_validate_image.feature Outdated
@fullsend-ai-review

fullsend-ai-review Bot commented Jul 14, 2026

Copy link
Copy Markdown

Review — approve

Tested: pin-policy-bundle acceptance coverage for verify-conforma-konflux-ta task

Summary

This PR adds acceptance test coverage for the pin-policy-bundle step in the TA task variant, closing a gap where this task had zero dedicated coverage for that step. The change correctly mirrors the two scenarios already covered by the verify-enterprise-contract task:

  1. No-op path — The existing "Golden container image with trusted artifacts" scenario now asserts the pin-policy-bundle step log. Since this scenario's policy uses git sources (not OCI), the step correctly reports 'oci::quay.io/conforma/release-policy:konflux' not found in policy configuration, nothing to do.

  2. Replacement path — A new "Pin policy bundle digest" scenario uses an OCI policy source and passes POLICY_BUNDLE_DIGEST with a digest (sha256:2a851b...) that is intentionally distinct from the task's default (sha256:f9049...). This ensures the snapshot comparison confirms the supplied value is actually propagated, rather than falling through to the default.

Checks performed

Dimension Finding
Correctness Scenarios correctly exercise both code paths of pin-konflux-policy-bundle.sh. Snapshot content matches expected script output. All TA-specific parameters (SNAPSHOT_FILENAME, SOURCE_DATA_ARTIFACT, TRUSTED_ARTIFACTS_DEBUG, ORAS_OPTIONS) are properly included.
Security No concerns — uses existing test public keys and golden container image references already present in the repository.
Intent & coherence Well-scoped test-only change that fills a documented coverage gap. Follows established patterns from the non-TA task tests.
Style/conventions Gherkin step alignment, comment style, and snapshot formatting are consistent with existing scenarios.
Documentation No docs changes needed for test-only additions.
Cross-repo contracts No production code or API surface changes.

No findings.

Previous run

Review

Outcome: Approve ✅

Summary

This PR adds pin-policy-bundle acceptance test coverage for the verify-conforma-konflux-ta (Trusted Artifacts) task, mirroring the two scenarios already covered by the verify-enterprise-contract task. The change is well-structured, correctly follows existing patterns, and closes a real coverage gap.

Analysis

Correctness

  • The no-op path (added assertion in "Golden container image with trusted artifacts") is correct: the policy uses git:: sources, so pin-policy-bundle finds no matching oci::quay.io/conforma/release-policy:konflux reference and reports "nothing to do." This exactly mirrors the EC task's Golden scenario behavior.
  • The replacement path (new "Pin policy bundle digest" scenario) correctly uses an OCI-based policy source that matches the pin target, verifying the full tag→digest replacement flow.
  • Snapshots are identical to their EC task counterparts, which is expected since the underlying pin-policy-bundle step implementation is shared.
  • STRICT=false is appropriate for the replacement path test — the test only needs to verify pinning mechanics, not full policy evaluation.
  • TA-specific parameters (SNAPSHOT_FILENAME, SOURCE_DATA_ARTIFACT, TRUSTED_ARTIFACTS_DEBUG, ORAS_OPTIONS) are correctly included, consistent with all other TA scenarios in the file.
  • The POLICY_BUNDLE_DIGEST value (sha256:f904979d...) matches the default in the task definition at tasks/verify-conforma-konflux-ta/0.1/verify-conforma-konflux-ta.yaml:174.

Security — No concerns. Test-only change using existing public test fixtures and keys.

Intent & coherence — The PR description accurately describes the changes. The diff matches the stated intent. No scope creep.

Style & conventions — Feature file structure, indentation, snapshot naming conventions, and assertion patterns all follow existing precedent exactly.

Documentation — No documentation updates needed for a test addition.

Cross-repo contracts — Pure test addition with no API, schema, or interface impact.


Labels: PR adds acceptance test scenarios for pin-policy-bundle coverage

fullsend-ai-review[bot]

This comment was marked as outdated.

@fullsend-ai-review fullsend-ai-review Bot added ready-for-merge All reviewers approved — ready to merge testing labels Jul 14, 2026
simonbaird
simonbaird previously approved these changes Jul 14, 2026
The scenario was using the same digest as the task default, so it
couldn't distinguish explicit parameter propagation from fallthrough
to the default. Use a clearly different test digest so the snapshot
comparison actually confirms the supplied value is used.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@fullsend-ai-review

fullsend-ai-review Bot commented Jul 14, 2026

Copy link
Copy Markdown

🤖 Finished Review · ✅ Success · Started 8:56 PM UTC · Completed 9:00 PM UTC
Commit: 87c4a29 · View workflow run →

@fullsend-ai-review fullsend-ai-review Bot added ready-for-merge All reviewers approved — ready to merge and removed ready-for-merge All reviewers approved — ready to merge labels Jul 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready-for-merge All reviewers approved — ready to merge size: M testing

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants