Skip to content

feat(core): add percentile pooling methods (median, perc5, perc10, perc20) (ADR-1181) - #1392

Draft
lusoris wants to merge 10 commits into
masterfrom
feat/percentile-pooling-methods
Draft

feat(core): add percentile pooling methods (median, perc5, perc10, perc20) (ADR-1181)#1392
lusoris wants to merge 10 commits into
masterfrom
feat/percentile-pooling-methods

Conversation

@lusoris

@lusoris lusoris commented Sep 7, 2026

Copy link
Copy Markdown
Contributor

Summary

Implements upstream Netflix/vmaf#818: VMAF_POOL_METHOD_MEDIAN, PERC5, PERC10
and PERC20, evaluated by sort plus linear rank interpolation (matching NumPy's
default percentile).

This is the pooling half of #1311, split out. That PR bundled ADR-1181
(percentile pooling) with ADR-1182 (Windows UTF-8 path contract). The two are
independent, and bundling them dragged model_loader.c, vmaf_roi.c and
vmaf_per_shot.c into the touched-file set purely to route their fopen calls
through the shim — three files carrying ~71 pre-existing clang-tidy findings that
rule 12 then made this PR's problem. The UTF-8 shim follows separately.

⚠️ Breaking: the emitted output schema widens

Both writers loop for (j = 1; j < VMAF_POOL_METHOD_NB; j++), so extending the
enum adds an attribute per method to every <metric> element and every JSON
pool object:

<!-- before -->
<metric name="psnr_y" min="29.640688" max="34.760779" mean="30.755064" harmonic_mean="30.727905" />
<!-- after -->
<metric name="psnr_y" min="29.640688" max="34.760779" mean="30.755064" harmonic_mean="30.727905"
        median="30.526752" perc5="29.843079" perc10="29.903105" perc20="30.025991" />

There is no flag to select a subset. The four pre-existing values do not move
verified byte-identical on the 576x324 pair with and without frame skipping — so a
reader that looks an attribute up by name is unaffected; one that matches a
<metric> line exactly, or rejects unknown attribute names, must be updated.

python/test/command_line_test.py carried three such exact-match assertions and is
updated here. The scores in them are unchanged — only the four new attributes
are appended. That file is not one of the five Netflix golden-assertion files in
CLAUDE.md §8, and no assertAlmostEqual value is touched anywhere in this PR.

Also in here

  • output.cpp is now clang-tidy clean (29 findings → 0). Most were style, but
    8 were bugprone-suspicious-stringview-data-usage: fmt_or_default() returned a
    std::string_view whose .data() went straight to std::fprintf as a format
    string
    . string_view::data() carries no null-termination guarantee, so that is
    UB the moment the view is built from a non-terminated buffer. It returns
    const char * now — latent rather than live, since today's callers all pass
    terminated data, but worth removing while the file is open.
  • test_pooling_percentile skips instead of failing without its fixtures. The
    Netflix golden YUVs are untracked (scripts/test/fetch-test-yuvs.sh fetches
    them), and only the golden-harness job restores them — so the sanitizer, coverage
    and ARM legs had no file to open. It now probes and sets mu_skipped (exit 77).
    It also no longer leaks its model and context on early returns, which under
    LeakSanitizer had been turning one failed assertion into 96912 bytes of unrelated-
    looking LSan output.

Reproducer / smoke-test command

meson setup core/build core -Denable_cuda=false -Denable_sycl=false -Denable_float=true
ninja -C core/build && meson test -C core/build --suite=fast

# The widened schema, and that the original four values are unmoved:
Y=python/test/resource/yuv
core/build/tools/vmaf --reference $Y/src01_hrc00_576x324.yuv \
  --distorted $Y/src01_hrc01_576x324.yuv --width 576 --height 324 \
  --pixel_format 420 --bitdepth 8 --xml --feature psnr \
  --model path=model/other_models/vmaf_v0.6.0.json --quiet --output /tmp/o.xml
grep -o '<metric name="psnr_y"[^/]*/>' /tmp/o.xml

# output.cpp is clean:
clang-tidy -p core/build --quiet core/src/output.cpp | grep -c warning:   # 0

Measured: meson test --suite=fast 117 Ok / 0 Fail; clang-tidy on output.cpp
29 → 0.

Deep-dive deliverables (ADR-0108)

  • Research digest — no digest needed: the algorithm is specified in ADR-1181 and the upstream issue; this PR implements it.
  • Decision matrixADR-1181 ## Alternatives considered.
  • AGENTS.md invariant note — no rebase-sensitive invariants beyond the rebase note below, which carries all four.
  • Reproducer / smoke-test command — above.
  • Changelog fragmentchangelog.d/added/pooling-percentile-methods.md, with the breaking schema change called out; CHANGELOG.md regenerated.
  • Rebase notedocs/rebase-notes.md § ADR-1181: VMAF_POOL_METHOD_NB is not a stable ABI count, extending the enum widens the output schema, the percentile family must bypass the streaming accumulators, and the fixture skip must not become a hard assert.

Docs (rule 10)

docs/usage/cli.md documents the new pooled attributes and
tells readers to tolerate new attribute names;
docs/api/index.md covers the enum additions.

State (rule 13)

docs/state.md: T-UPSTREAM-818-POOLING-ENUM-NO-PERCENTILES-2026-09-03 moves from
Open bugs to Recently closed, citing this PR.

🤖 Generated with Claude Code

@lusoris
lusoris force-pushed the feat/percentile-pooling-methods branch from a3f3b00 to 10d8914 Compare September 7, 2026 13:28
@lusoris
lusoris marked this pull request as ready for review September 7, 2026 14:25
@lusoris
lusoris enabled auto-merge (squash) September 7, 2026 14:25
@lusoris

lusoris commented Sep 7, 2026

Copy link
Copy Markdown
Contributor Author

Blocked on Tidy Ratchet, and the two tidy gates disagree with each other on the same file in the same CI run.

tidy-ratchet[cpu]: 289 TUs, 2922 warnings (baseline 2950), 58 uncited NOLINTs (baseline 58)
::error core/src/output.cpp: warnings 29 -> 0 (-29) — tighten the baseline
::error core/test/test_pooling_percentile.c: warnings 0 -> 1 (+1) — fix the code, never raise the baseline

The output.cpp half is expected and good — this PR took that file to zero. The +1 is the problem, and I cannot see the diagnostic:

Gate Same runner, same commit 036005ca0 Result for test_pooling_percentile.c
Tidy Changed clang-tidy-22 -p build --quiet <changed files>, fails on any warning SUCCESS — 0 warnings
Tidy Ratchet clang-tidy -p build <source>, counts per file 1 warning

The file is not in Tidy Changed's exclusion list, so that pass is not vacuous — it genuinely linted it and found nothing.

What I ruled out locally, replicating the ratchet's exact invocation (cwd = build dir, source path as it appears in compile_commands.json, -Denable_cuda=false -Denable_sycl=false -Db_lto=false, clang-tidy 22.1.8 — the same version the report records):

  • gcc 16.2.1 build → 0 warnings attributed to the file
  • gcc-15 build → 0 warnings attributed to the file
  • both: stdout empty, stderr only 1 warning generated. Suppressed 29 warnings (1 in non-user code, 28 NOLINT). — i.e. the one warning is a system-header diagnostic (glibc stdlib.h, bugprone-casting-through-void), which parse_diagnostics correctly drops via relpath returning None for out-of-repo paths

CI builds the ratchet lane with CC=gcc-14 CXX=g++-14; I have 15 and 16 only. The most likely explanation is a diagnostic that lands inside the test TU under gcc-14's headers and inside a system header under 15/16 — but that is a hypothesis, not something I have evidence for.

Why I am not just baselining it: the ratchet explicitly refuses ("fix the code, never raise the baseline"), and it is right to.

What would settle it: the ratchet's report artifact records per-file counts only, not the diagnostics. Having tidy-ratchet.py emit the matched diagnostic lines into the report — it already parses them in parse_diagnostics — would turn this from an unreproducible disagreement into a one-line answer. That looks worth doing on its own merits, since any future +1 on a machine that cannot reproduce CI's toolchain hits exactly this wall.

Holding in the merge train meanwhile; it has held the window ~25 minutes.

@lusoris
lusoris marked this pull request as draft September 7, 2026 14:56
auto-merge was automatically disabled September 7, 2026 14:56

Pull request was converted to draft

lusoris pushed a commit that referenced this pull request Sep 7, 2026
The clang-tidy lanes install three toolchain components and treated them
inconsistently. clang comes from apt.llvm.org at 22 because ubuntu-24.04 ships
18, which cannot parse the tree's C++26. meson comes from PyPI because the
distro's 1.3.2 predates `c23` in c_std. gcc was left at the image's gcc-14,
with no rationale in the workflow and no ADR pinning it -- the only mention of
gcc-14 in docs/adr/ is an incidental line in a VVenC changelog summary.

That stopped being cosmetic on PR #1392. The ratchet reported
test_pooling_percentile.c going 0 -> 1 while Tidy Changed -- which fails on ANY
warning and does not exclude that file -- called the same file clean, on the
same runner and commit. It could not be reproduced on gcc-15 or gcc-16.

The cause is structural: clang-tidy parses each TU against the system headers
the C compiler provides, so a ratchet count depends on gcc's version as much as
on clang-tidy's, and the baseline recorded only clang_tidy_version. Pinning gcc
a major behind every developer's box while recording nothing about it makes a
disagreement unexplainable.

Three changes:

  - the lanes install gcc-15/g++-15 from ppa:ubuntu-toolchain-r/test;
  - the ratchet records cc_version, read from the build directory's
    meson-info/intro-compilers.json so it is the compiler that actually
    produced compile_commands.json, and annotates a mismatch like it already
    does for clang-tidy;
  - the --report artifact keeps every diagnostic as `path:line:col: [check]`.
    parse_diagnostics() already produced them and the script discarded all but
    the count. The baseline stays counts-only so it remains reviewable and does
    not churn on line-number shifts.

Verified locally: the report now carries `cc (GCC) 16.2.1 20260810` and 29
diagnostics for output.cpp, naming check and location.

This requires a CI-side regeneration of tidy-baseline-cpu.json -- counts under
gcc-15's headers will not equal gcc-14's, and a local --write would resolve
enable_dnn=auto differently and rebaseline unrelated files against the wrong
build.

libvmaf-build-matrix.yml still pins gcc-14; moving it surfaces its own crop of
warnings across ten-plus legs and belongs in its own PR.

Refs: ADR-1230

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@lusoris
lusoris force-pushed the feat/percentile-pooling-methods branch 2 times, most recently from 10add61 to f4eea16 Compare September 7, 2026 17:02
Lusoris and others added 10 commits September 7, 2026 20:35
…rk introduced

ADR-0141 / ADR-1142 require every file a PR touches to stay at or below its
committed clang-tidy baseline. The percentile-pooling and UTF-8-path commits
added five findings to `core/src/libvmaf.c` (which is at zero on master) and
introduced three new translation units carrying findings of their own.

- `libvmaf.c`: split the ADR-1181 percentile path out of
  `vmaf_feature_score_pooled()` into `pool_percentile()` plus two small
  helpers, which clears `readability-function-size` (92 lines / 21 branches)
  and the three `readability-braces-around-statements` findings from the
  `perc` if/else chain. Dropped the four percentile `case` labels in
  `pool_reduce()` that returned the same `-EINVAL` as `default:`
  (`bugprone-branch-clone`); the explanation moved onto `default:`.
- `compat/path_utf8.c`, `test/test_path_utf8.c`, `test/test_pooling_percentile.c`:
  ADR-1138 `NOLINTBEGIN(modernize-use-nullptr)` block with the standing
  citation, matching every other C translation unit in the tree.
- `test_path_utf8.c`: split the three oversized test bodies into
  write / read-back / error-class helpers so none exceeds the 15-branch budget,
  moved the Windows wide-API cross-check behind `assert_wide_path_exists()`,
  closed the write and read handles before asserting on the transfer result,
  and cited ADR-1143 on the single-threaded `getenv` test setup.
- `test_pooling_percentile.c`: split the end-to-end body into
  `feed_reference_pair()` / `check_percentile_oracles()` /
  `check_central_oracles()` / `check_pooled_error_paths()`, which also closes
  the `clang-analyzer-unix.Stream` leak (both YUV handles were left open when
  an assertion between `fopen` and `fclose` returned) and the
  `readability-isolate-declaration` finding. No oracle value changed.

Also documents the output-schema half of ADR-1181 in `docs/usage/cli.md`:
every `<metric>` row and JSON object now carries `median`, `perc5`, `perc10`
and `perc20`, with the measured values for the documented 576x324 pair.

Verified: `meson test -C build-prep --suite=fast` 116/116, and `clang-tidy -p
build-prep` reports zero findings on `path_utf8.c`, `libvmaf.c`,
`test_path_utf8.c` and `test_pooling_percentile.c`.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
An earlier keep-both rebase resolution left T-UPSTREAM-818-POOLING-ENUM-NO-PERCENTILES-2026-09-03
duplicated, and the follow-up dedup commit dropped the wrong copy: it kept
master's Open row and deleted this branch's Recently-closed row. Rebasing onto
current master then removed the Open row as well (master still carried it,
this branch closes it), leaving the id with no row at all — an ADR-0165
violation that scripts/ci/check-state-md-rows.sh cannot see, because it only
rejects duplicates.

Restores the Recently-closed row, corrects its algorithm description (the
implementation sorts with qsort and interpolates linearly between neighbouring
ranks, matching NumPy's default percentile; it is not quickselect), and moves
both closed rows from the branch name to `PR #1311` in the provenance column,
matching the convention the other 2026-09-06 rows use.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
My rebase dedup kept the wrong side. Its rule -- when a bug id appears twice
after a keep-both rebase, keep the row matching origin/master -- is right when
the branch merely restates a row master already carries, and WRONG when the
branch is the PR that closes the bug. Then the branch's past-tense
Recently-closed row is the correct survivor and master's present-tense Open row
is the stale one.

Restored the closure row and left a tombstone where the Open row was, matching
the convention used elsewhere in this file. Exactly one row per bug id;
scripts/ci/check-state-md-rows.sh reports OK.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… absent

test_pooling_percentile is registered in the `fast` suite and opens the
Netflix 576x324 golden pair through a hard mu_assert. Those YUVs are
deliberately untracked -- .gitignore keeps them out of the tree because of
their size and scripts/test/fetch-test-yuvs.sh fetches them -- and only the
golden-harness job restores them from cache. Every other job that runs the C
unit tests therefore had no fixture to open: Sanitizers (address, thread,
undefined), Sanitizers ASan+UBSan, Coverage Gate and Ubuntu ARM clang all
failed on this one test.

Probe for the pair first and set mu_skipped when it is missing, so the
harness exits 77 and meson reports "skipped". "The fixture is not here" and
"the pooling maths is wrong" are different facts and must not share an exit
code.

Second defect, which is what turned the failure into a confusing one: every
early return in test_pooling_percentile_yuv leaked the model and the context.
Under LeakSanitizer that converted a single failed assertion into 96912 bytes
of LSan output in 33 allocations, whose stacks pointed at vmaf_model_load and
feature_collector_mount_model rather than at the missing file. The checks now
funnel through one exit that always calls vmaf_close and
vmaf_model_destroy.

Verified under -Db_sanitize=address,undefined, both paths:

  fixtures absent  -> "[skip: Netflix golden YUVs not fetched]", exit 77,
                      no leaks
  fixtures present -> 2 tests run, 2 passed, no leaks

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
ADR-1182's UTF-8 path contract and ADR-1181's percentile pooling arrived as
one PR (#1311). They are independent, and bundling them dragged
model_loader.c, vmaf_roi.c and vmaf_per_shot.c into the PR's touched-file set
purely to route their fopen calls through the shim. Those three files carry
~71 pre-existing clang-tidy findings between them, which rule 12 then makes
this PR's problem.

Keep the pooling half here; the shim goes out separately.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ted format

Tidy Changed lints every file a PR touches and fails on any warning, so
output.cpp's 29 pre-existing findings blocked this PR. Discharge them:

  15x misc-use-anonymous-namespace + 1x misc-use-internal-linkage — the file's
  helpers and the LocaleGuard class move into two anonymous namespaces (the
  public vmaf_write_output_* entry points sit between the helper groups, so it
  cannot be one region) and drop their `static`.

  8x bugprone-suspicious-stringview-data-usage — fmt_or_default() returned a
  std::string_view whose .data() was handed straight to std::fprintf as a
  format string. string_view::data() carries no null-termination guarantee, so
  that is undefined behaviour the moment the view is ever built from a
  non-terminated buffer. It returns const char* now, which keeps the guarantee
  in the type rather than in a comment. Not merely a lint: the current callers
  happen to pass null-terminated data, so this was latent rather than live.

  4x misc-const-correctness, 1x readability-braces-around-statements.

Verified: clang-tidy-22 on output.cpp goes 29 warnings -> 0, the CPU build is
clean, and meson test --suite=fast reports 118 Ok / 0 Fail.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Tidy Changed lints headers as their own translation unit, where nothing has
pulled stddef.h in yet, so the three assert(x != NULL) uses reported
'use of undeclared identifier NULL' as clang-diagnostic-error.

The header compiled fine in every real build only because every includer
happened to have stddef.h already. That is the definition of a header that is
not self-contained, and it breaks the moment someone includes it first.

clang-tidy on the header standalone: 0 findings. Build clean,
meson test --suite=fast 117 Ok / 0 Fail.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@lusoris
lusoris force-pushed the feat/percentile-pooling-methods branch from f4eea16 to fa92170 Compare September 7, 2026 18:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant