fix(fetch/alma-errata): restore advisories dropped by upstream#191
Open
shino wants to merge 4 commits into
Open
fix(fetch/alma-errata): restore advisories dropped by upstream#191shino wants to merge 4 commits into
shino wants to merge 4 commits into
Conversation
…ed re-curations AlmaLinux re-curated the AlmaLinux 10 errata.full.json feed twice (2025-11-26: -102 advisories, 2026-05-27: -143 advisories / ~376 CVEs), dropping published advisories from the feed while their HTML pages and the official OSV export still carry them (asked upstream which side is authoritative: https://bugs.almalinux.org/view.php?id=644). Split the alma-errata fetch out of fetch-main into its own workflow, following the fetch-paloalto-json-or-csaf pattern: after the fetch, any file still carried by a restore source (pre-fetch HEAD + the parents of the two purge commits) but missing from the fresh fetch is restored, so the raw repository keeps the union of everything upstream ever published, matching the HTML/OSV semantics. Verified locally: restoring 143+102 files on top of the current feed yields exactly the 511-advisory set the OSV export carries for AlmaLinux 10. Unlike paloalto-json there is no upstream list to validate against, so there is no allowlist and upstream deletions never propagate on their own; the step emits a warning whenever it restores anything so new re-curations stay visible. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Pull request overview
This PR updates the CI fetching pipeline to make AlmaLinux Errata ingestion resilient to upstream feed “re-curations” that silently drop already-published advisories, by introducing a dedicated fetch-alma-errata workflow that restores missing advisory JSON files from pinned historical snapshots.
Changes:
- Removed
alma-erratafrom the genericfetch-mainworkflow dispatch options and from thefetch-allmatrix. - Added a new reusable workflow
fetch-alma-errata.ymlthat fetches the current feed and then restores any dropped.jsonadvisories fromHEADand two pinned snapshot commits, failing hard if those snapshots are missing from history. - Wired
fetch-all.ymlto run the new Alma errata fetch as a separate job and include it in the overallcheckgating.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
| .github/workflows/fetch-main.yml | Removes alma-errata from the workflow_dispatch target choice list so it’s no longer run via the generic fetch workflow. |
| .github/workflows/fetch-alma-errata.yml | New dedicated workflow that fetches Alma errata and restores advisories dropped by upstream from pinned history sources before committing/pushing. |
| .github/workflows/fetch-all.yml | Removes alma-errata from the main fetch matrix and adds a standalone fetch-alma-errata job included in downstream needs. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
…t path list Replace the loose restore-everything-missing behavior with an explicit 245-path RESTORE_PATHS list (the exact union the two re-curations dropped; current feed + these = the 511-advisory OSV set, verified) and turn every other disappearance into a hard failure: a file carried by the pre-fetch HEAD but missing from the fresh fetch, and in neither RESTORE_PATHS nor ACCEPTED_REMOVAL_PATHS, fails the job for human triage, mirroring the paloalto-json allowlist flow. Entries the feed carries again emit a warning to prune the list. Simulated against the real dotgit: 0 unexpected drops, 245 restored, resulting AlmaLinux 10 set diff-equal to the osv.dev export. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…L index Replace the manual ACCEPTED_REMOVAL_PATHS escape hatch with an automatic oracle: the per-version HTML directory index on errata.almalinux.org (one GET per version, zero requests when nothing dropped). A feed drop whose HTML page is still published is a re-curation and fails the job for RESTORE_PATHS triage as before; a drop whose HTML page is gone is a genuine upstream retirement and propagates with a warning. RESTORE_PATHS entries are held to the same standard: an entry about to be restored whose HTML page has vanished fails the job until a human removes it, so the pinned list can never resurrect something upstream fully retired. An unreachable or zero-entry index fails the job rather than guessing; the Fetch step already depends on the same host, so this adds no new availability dependency. Simulated against the real dotgit and live index (511 pages parsed): normal run restores 245; an unlisted drop with a live HTML page fails; a listed entry without an HTML page fails; a drop without an HTML page propagates with a warning while the remaining 245 restore. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Replace the ordered RESTORE_SNAPSHOTS search (first snapshot carrying a file wins) with explicit RESTORE_SNAPSHOT_N / RESTORE_PATHS_N pairs: each of the 245 paths now declares the exact commit it is restored from (143 from 8255896a, 102 from 593d3bca), so provenance is auditable in the diff and future re-curations append a self-contained pair instead of relying on iteration order. A path listed under a snapshot that does not carry it, or listed in more than one block, fails the job. Simulated against the real dotgit and live index: normal run restores 245 (143+102); a misassigned entry and a duplicate entry each fail; the HTML-index arbitration scenarios behave unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
shino
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
errata.full.jsonfeed twice, silently dropping published advisories while their HTML pages (https://errata.almalinux.org/10/) and the official OSV export (AlmaLinux/osv-database, osv.dev) keep carrying them:cb5d6bc44(raw, 2025-11-26): -102 advisories (the ALSA-2025:7457..19772 range + ALSA-2025:A003/A006)e64c5ca88(raw, 2026-05-27): -143 advisories, ~376 CVEs losing detection coverage — the event that forced the current alma-errata pin indb-main.mk/db-nightly.mkfetch-maininto its own workflow following thefetch-paloalto-json-or-csaf.ymlpattern: after the fetch rewrites the tree from the current feed, restore exactly the 245 explicitly listed advisories, each from the pinned pre-purge snapshot its list block declares (RESTORE_SNAPSHOT_N/RESTORE_PATHS_Npairs: 143 from8255896a, 102 from593d3bca— the parents of the two purge commits). Nothing unlisted is ever restored, per-file provenance is explicit in the diff, and a misassigned or duplicated entry fails the job.RESTORE_PATHS_Nblock::warning::::warning::prompting list pruning.Verification
actionlintclean on the new/changed workflows (the two pre-existing SC2086 infos infetch-all.yml'scheckjob are untouched).vuls-data-raw-alma-erratadotgit and the live HTML index (511 pages parsed):8255896a, 102 from593d3bca); resulting AlmaLinux 10 set (511) matches both the osv.dev export and the HTML index exactlyRESTORE_PATHS_Nblock → fails as a duplicate<version>/ALSA/<year>/<ID>.json), so restored files stay under fetch management.Follow-up (separate PR)
Once this has run and
extract alma-erratahas picked the restored advisories up, unpin alma-errata indb-main.mk/db-nightly.mk. The first unpinned run will trip diff-guard on alma_10/alma:10 by design (baseline jump) and needs the usual promote.🤖 Generated with Claude Code