bench: add the Netflix Chimera BarScene corpus entry, and gate grain_sigma on grain_flat - #49
Merged
Merged
Conversation
…sigma on grain_flat The bbb pin is clean animation, and ADR-0142's own measured law says reductive pre-encode gain scales with removable impairment -- ~0 on clean. Benchmarking the core thesis mainly on clean animation cannot really falsify it either way. Add netflix-bar: Netflix's Chimera BarScene, donated to the Xiph.Org derf collection. Real camera content, measured at the identical 640x360/48-frame workload: 2.4x the texture, 3.1x the variance, 1.7x the banding of bbb. Also a foundation-run host rather than the single third party bbb now depends on. It is deliberately documented as a COMPLEXITY entry, not a grain one. I assumed real camera content would be grainier and measured instead: grain_sigma 0.0124 vs bbb's 0.0131 -- marginally LOWER. The pinned .webm is a VP9 distribution copy and the encode removed the grain; the ungraded .y4m retains it but is 29.6 GB. For a grain axis, run-bench.py --synth noise already exists (and is deterministic -- checked, ffmpeg's noise filter has a fixed default seed). While measuring, found something that matters for the unbuilt tune=auto router: the estimator accumulates only over flat neighbourhoods, so grain_flat is the COVERAGE of the sigma estimate. On real content the response is correctly monotonic (sigma 0.0132 -> 0.0214, flat 0.331 -> 0.005 as injected noise goes 0 -> 20), but on a pathologically flat source almost nothing qualifies, flat collapses to ~0 and sigma becomes 0 or unstable. Keyed on sigma alone, a starved estimate reads as 'clean' and would route heavy-grain content AWAY from denoise -- inverting the case with the largest measured win. Documented, with the table. This is a consumer-side gating rule, not a filter defect: the estimator is correct on real content. Also fix a latent parser bug in fetch-corpus.sh: it ran xargs on every line BEFORE skipping comments, so an apostrophe in a corpus.lock comment killed the whole fetch with 'unmatched single quote'. Verified: cold fetch of all three entries downloads, sha256-matches and extracts; both real clips yield the identical 16,588,800-byte workload. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The
bbbpin is clean animation. ADR-0142's own measured law is that reductive pre-encode gain scales with removable impairment — ~0 on clean. So benchmarking the core thesis mainly on clean animation can't really falsify it either way.What
netflix-bar— Netflix's Chimera BarScene, donated to the Xiph.Org derf collection. Real camera content, measured at the identical 640x360/48-frame workload:bbbnetflix-barmedia.xiph.org is also a foundation-run archive, rather than the single third party
bbbnow depends on after blender.org went dead.It is NOT a grain entry — I assumed it would be, and measured otherwise
grain_sigmais 0.0124 vs bbb's 0.0131 — marginally lower. The pinned.webmis a VP9 distribution copy (~6.9 Mbit/s for 4K) and the encode removed the grain. The ungraded.y4mretains it but is 29.6 GB, which is not pinnable. Documented as a complexity entry so nobody repeats the assumption.For a grain axis,
run-bench.py --synth noisealready exists — and is deterministic (checked: ffmpeg's noise filter has a fixed default seed, so the missingall_seedis not a reproducibility bug).Found while measuring:
grain_sigmaneedsgrain_flatas a gateThe estimator accumulates only over flat neighbourhoods (
if ((hi-lo) > edge_thr) return;), sograin_flatis the coverage — and therefore the confidence — of the sigma estimate.On real content the response is correctly monotonic in both:
grain_sigmagrain_flatBut on a pathologically flat source (uniform colour or smooth gradient, uniformly noised) almost nothing qualifies,
grain_flatcollapses to ~0, and sigma becomes 0 or unstable.This matters for the unbuilt
tune=autorouter (ADR-0142), which usesgrain_sigmaas its grain-detection input: keyed on sigma alone, a starved estimate reads as clean and would route heavy-grain content away from the denoise leg — the exact case with the largest measured BD-rate win. Now documented with the disambiguation table.To be clear this is a consumer-side gating rule, not a filter defect — the estimator is correct on real content. I nearly filed it as a bug off the synthetic result before testing real content.
Also
Latent parser bug in
fetch-corpus.sh: it ranxargson every line before skipping comments, so an apostrophe in acorpus.lockcomment killed the whole fetch withunmatched single quote.Verified
Cold fetch of all three entries downloads, sha256-matches and extracts; both real clips yield the identical 16,588,800-byte workload. 21/21 fast suite, changelog in sync.
🤖 Generated with Claude Code