Skip to content

codecov/patch is unconfigured and permanently red — split genuine gaps from differential-covered lines (v0.56) #923

Description

@avrabe

Deferred to v0.56 by decision during the v0.55 assembly.

The problem

codecov/patch fails on essentially every PR and has been treated as advisory for dozens of merges. That habit is not free: it is exactly the reflex that once let a red Clippy through as "just codecov" (#683, main went red).

Three facts, measured 2026-08-07:

  1. There is no codecov.yml anywhere in the repo. The check runs on codecov's defaults — patch target auto, i.e. changed lines must meet the project average. Nobody chose that threshold.
  2. The coverage JOB passes. Rust-test Line Coverage (unit + integration only)success. Only codecov's external patch status is red.
  3. VG-009 (shipped this release) explains much of it. cargo llvm-cov instruments the Rust test suite IN-PROCESS; the execution differentials spawn synth as a separate, uninstrumented process, from other CI jobs, so they emit no profile data at all. synth-backend-aarch64/src/backend.rs reads 41.6 % while being exercised end-to-end constantly.

So an unknown fraction of "untested" lines are executed against wasmtime hundreds of times per run — just not where the instrument can see them. Writing unit tests for those raises a number without adding evidence.

Why it still matters

The rest of the flagged lines genuinely ARE untested, and today they are indistinguishable from the differential-covered ones. That is the same defect shape as #921's unmodeled-op: a signal that says something is wrong but not which, so the only response is to ignore it — and an ignored red is a gate that cannot do its job ([[gate-potency]]: a check everyone is trained to ignore is worse than no check).

Plan

  1. Measure the split. Run llvm-cov on a failing lane and classify every flagged line: genuinely-uncovered vs covered-only-by-a-differential. Data, not opinion — this is the step that decides the rest.
  2. Write the tests the split says are real. Worth having on their own merits.
  3. Add codecov.yml stating a threshold someone actually chose, and mark the patch status informational: true EXPLICITLY rather than de-facto. If it is advisory, say so in config so a future reader is not left inferring it from merge habits.

Related: #910 (what a wired oracle attests), VG-009/VG-010 in artifacts/verification-gaps.yaml, #683 (the merge this noise contributed to).

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions