Skip to content

feat: fail closed on unverified head snapshot in dependency-vuln-check#749

Merged
szpraat merged 2 commits into
mainfrom
feat/vuln-gate-head-snapshot-guard
Jul 14, 2026
Merged

feat: fail closed on unverified head snapshot in dependency-vuln-check#749
szpraat merged 2 commits into
mainfrom
feat/vuln-gate-head-snapshot-guard

Conversation

@szpraat

@szpraat szpraat commented Jul 13, 2026

Copy link
Copy Markdown
Contributor

What

Adds a head-side verification guard to dependency-vuln-check, mirroring the fail-closed treatment the base side already has.

The gate diffs compare(effective_base ... head_sha). The base side is resolved to a verified snapshotted ancestor and fails closed when it can't be established. The head side had no equivalent guard — check.py trusted the compare result and failed closed only if the API call itself errored. It never verified the head SBOM was actually submitted and indexed. So:

  • if the head-SBOM submission job failed, or
  • if the submission hadn't finished indexing when the compare ran,

the head side read empty and a genuinely new vulnerable dependency passed silently — a false negative. Harmless while the consuming gate was observe-only; under blocking it's a false green.

Two guards

  1. Submission failure — new optional input head-snapshot-succeeded. The consumer passes the result of its head-SBOM submission job (e.g. needs.pr-maven-snapshot.result == 'success'). When 'false', the gate fails closed (honoring the ci:vuln-gate-override label, same as the base-side paths). Unset → skipped, so existing consumers are unaffected.
  2. Indexing lag — when the head side of the diff looks empty (0 added, deps removed), the compare is re-fetched a few times with backoff to let the dependency graph settle. A removal-only PR has the identical shape and simply re-confirms — the gate never blocks on this signal, it only waits out lag, so it cannot misfire into a false positive.

Testing

68/68 pytest pass (was 60). 8 new tests: _looks_like_missing_head helper (3), gap-1 fail-closed + override-bypass + true-proceeds (3), gap-2 retry-then-block-on-settled-diff + persistent-empty-never-blocks (2). README + action inputs + fail-closed contract documented.

Rollout

Backward compatible (new input defaults to unset/skip). After merge, the consumer (camunda/camunda ci.yml) wires head-snapshot-succeeded and Renovate bumps the pin. Companion to the vuln-gate blocking flip camunda/camunda#57561.

The base side of the dependency diff is resolved to a verified snapshotted
ancestor and fails closed when it cannot be established. The head side had no
equivalent guard: check.py trusted compare(effective_base...head_sha) and only
failed closed if the API call itself errored — never verifying the head SBOM
was actually submitted and indexed. If the head snapshot submission failed, or
had not finished indexing when the compare ran, the head side of the diff read
empty and a genuinely new vulnerable dependency passed silently (false
negative). Harmless while the consuming gate was observe-only; under blocking it
is a false green.

Two guards close it:

- Submission failure — new optional `head-snapshot-succeeded` input. The
  consumer passes the result of its head-SBOM submission job; when 'false' the
  gate fails closed (honors the override label, like the base-side paths).
  Unset preserves current behavior for existing consumers.
- Indexing lag — when the head side of the diff looks empty (0 added, deps
  removed) the compare is re-fetched a few times with backoff to let the
  dependency graph settle. A removal-only PR has the same shape and simply
  re-confirms; the gate never blocks on this signal, it only waits out lag, so
  it cannot misfire into a false positive.

68/68 pytest pass (8 new: helper, gap-1 fail-closed + override bypass, gap-2
retry-then-block and persistent-empty-never-blocks). README + action inputs
documented.
Copilot AI review requested due to automatic review settings July 13, 2026 08:44

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR strengthens dependency-vuln-check by adding head-side verification so the Dependency Review compare isn’t trusted when the PR head snapshot is missing/failed or still indexing, preventing false-negative “green” results when the gate becomes blocking.

Changes:

  • Adds optional head-snapshot-succeeded action input and a fail-closed path when it is explicitly 'false'.
  • Adds a head-diff “looks empty” heuristic with retry + backoff to absorb dependency-graph indexing lag before evaluating the final diff.
  • Adds unit tests for the new helper and head-guard behaviors, and updates README documentation.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

File Description
dependency-vuln-check/check.py Adds head snapshot guard, empty-head retry logic, and the _looks_like_missing_head helper.
dependency-vuln-check/action.yml Introduces head-snapshot-succeeded input and wires it into HEAD_SNAPSHOT_SUCCEEDED.
dependency-vuln-check/README.md Documents the new input and head-side verification/lag behavior.
dependency-vuln-check/test_check.py Adds tests covering submission-failed fail-closed behavior and indexing-lag retry behavior.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread dependency-vuln-check/check.py Outdated
Address Copilot review on #749: the guard blocked only on the literal 'false',
so a consumer wiring `needs.<job>.result` directly (yielding 'failure' /
'cancelled' / 'skipped') or a typo would silently be treated as "proceed" —
turning the guard into a no-op and reintroducing the false negative.

Now only '', 'true', 'success' proceed; every other value fails closed (the
submission result is named in the reason). A wiring mistake fails safe (blocks,
visible, override-able) instead of silently disabling the guard. README +
action input doc updated; 76/76 pytest (added unexpected-value + recognized-ok
parametrized tests).
@szpraat
szpraat requested a review from liliancavalet July 13, 2026 10:02
# not wire it) proceed; ANY other value — 'false', a raw job result like
# 'failure'/'cancelled'/'skipped', or a typo — fails closed, so a wiring
# mistake cannot silently turn this guard into a no-op.
if head_snapshot_ok not in ("", "true", "success"):

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

❓ The allow-list fix closes the failure / cancelled path nicely. One case it doesn't cover: success is still a "proceed" value, and the consumer's pr-maven-snapshot job is continue-on-error: true. If that masks a failed snapshot to needs.pr-maven-snapshot.result == 'success', the guard gets a legit-looking success and proceeds — the exact false negative this input is meant to catch.

Do we know what .result reports for a continue-on-error job whose step failed?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

needs.<job>.result returns success for a continue-on-error: true job whose step failed — the failure is masked and there's no job-level outcome/conclusion to read (actions/toolkit#1739). So wiring the guard to .result == 'success' would no-op it.

This isn't fixable in check.py. The guard receives a string; a masked failure and a true pass both arrive as the literal 'success'. It's well-formed, so the allow-list — which rejects failure/cancelled/typos — can't distinguish them. No logic here can.

The fix is at the source, in the consumer wiring: expose the step outcome as a job output and pass that instead of .result:

 pr-maven-snapshot:
   continue-on-error: true
   outputs: { submitted: "${{ steps.submit.outcome }}" }
   steps:
     - id: submit
       uses: advanced-security/maven-dependency-submission-action@...
 pr-vuln-check:
   with:
     head-snapshot-succeeded: ${{ needs.pr-maven-snapshot.outputs.submitted || 'failure' }}

steps.submit.outcome is pre-continue-on-error, so a failed step reads failure; || 'failure' covers the job dying before the step runs. I opened a dedicated PR on the consumer side

@szpraat
szpraat merged commit 1505e9e into main Jul 14, 2026
5 checks passed
@szpraat
szpraat deleted the feat/vuln-gate-head-snapshot-guard branch July 14, 2026 07:53
@infra-releases infra-releases Bot mentioned this pull request Jul 14, 2026
pull Bot pushed a commit to nagyist/camunda-zeebe that referenced this pull request Jul 14, 2026
camunda#57593)

## What

Wires the dependency vulnerability gate's `head-snapshot-succeeded`
guard to the **real submit-step outcome** instead of
`needs.pr-maven-snapshot.result`.

## Why

`pr-maven-snapshot` is `continue-on-error: true` (so a snapshot failure
falls back to auto-parse without failing the run). Job-level
`continue-on-error` masks a failed job to `needs.<job>.result ==
'success'` — there is no job-level outcome/conclusion to read
([actions/toolkit#1739](actions/toolkit#1739)).

So if the head-SBOM submission fails, `.result` still reads `success`.
Feeding that to the gate's `head-snapshot-succeeded` input hands it a
legit-looking success → the head side of the compare is stale/empty → a
genuinely new vulnerable dependency passes silently. Harmless while the
gate is observe-only; a **false green once it blocks**.

The gate guard itself (camunda/infra-global-github-actions#749) is
correct — it fails closed on any non-success value. It just cannot
distinguish a masked-failure `'success'` from a true one; both arrive as
the same string. The fix has to be at the source, in this consumer
wiring.

## Change

- `pr-maven-snapshot`: expose `outputs.submitted: ${{
steps.submit.outcome }}` (`id: submit` on the submit step). Step
`outcome` is pre-continue-on-error, so a failed step reads `failure`.
- `pr-vuln-check`: `head-snapshot-succeeded: ${{
needs.pr-maven-snapshot.outputs.submitted || 'failure' }}`. The `||
'failure'` coalesces an empty output (job died before the submit step
ran) to fail-closed.

## Ordering

- Input `head-snapshot-succeeded` lands with
camunda/infra-global-github-actions#749; it is inert (ignored) until
that action SHA is pinned here (Renovate), so this is safe to merge
before or after #749.
- **Must land before** the blocking flip (camunda#57561) — otherwise blocking
runs on an unguarded gate that can emit false greens with merge
authority.
@infra-releases infra-releases Bot mentioned this pull request Jul 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants