Skip to content

Verification gate: 'rivet validate' failed once, unreproducible — intermittent failure in a REQUIRED check #345

Description

@avrabe

A required gate failed once and passed on the identical tree. Filing so the next occurrence is not re-diagnosed from scratch.

What happened

On #340, the verification gate failed:

[   FAIL (rc=1)] (  0.97s) FV-FALCON-READINESS-001: rivet validate

Run: 31133692621 (job 92728227503), 2026-08-07 01:12Z. Duration 1h3m28s overall — well under the 90-minute timeout, so not a timeout.

Why it is worth filing

It did not reproduce. I checked, rather than assuming:

check result
rivet validate on the branch, pinned v0.19.0 PASS, exit 0 (108 warnings)
rivet validate on the exact PR merge commit d0c7140, pinned v0.19.0 PASS, exit 0
version CI used rivet-cli v0.19.0 … a251fbd8 — identical to the local build
same step, re-run after rebase (run 31139…, 15:02Z) PASS (0.76s)

Same tree, same binary, different outcome — so it is environmental or a race, not the content.

Why it matters

Verification gate (rivet-driven) is a required check. An intermittent failure in it randomly blocks merges and, worse, trains people to hit "re-run" on a red safety gate — which is exactly the habit that makes a real failure get clicked through.

Leads, not conclusions

  • The gate runs on self-hosted runners that reuse their workspace; leftover state between runs is the obvious suspect, though actions/checkout should clean.
  • The failing invocation took 0.97s vs 0.76s on the passing run — both fast, so it failed early rather than partway.
  • The gate only prints the last 15 lines on failure, and those were all gap warnings. The actual error line was above the cut, so the log did not contain the reason. If this recurs, capture the full rivet validate output — that is the single change that would make the next occurrence diagnosable.

Suggested first step

Widen the failure capture in run-falcon-verification.py from 15 lines to the full output (or upload it as an artifact) for steps that fail. Cheap, and it is the difference between "unreproducible" and "root-caused" next time.

🤖 Generated with Claude Code

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions