DRAFT: Rebase av2enc on main - #5331
Open
yeqing-wu wants to merge 205 commits into
Open
Conversation
See https://github.com/zizmorcore/zizmor. We pin GitHub Actions used in CI workflows to git hashes instead of version tags. Similar change was also made to libavif's CI workflows in PR AOMediaCodec/libavif#1515 at the suggestion of Google's security team. The rationale is described in issue AOMediaCodec/libavif#1514. <details> <summary><code>zizmor --fix=all --gh-token=$(gh auth token) .github/workflows</code> output</summary> ``` INFO zizmor: \U0001f308 zizmor v1.25.2 WARN collect_inputs: zizmor::registry::input: failed to validate file://.github/workflows/pull_request.yaml as workflow: input does not match expected validation schema WARN collect_inputs: zizmor::registry::input: failed to validate file://.github/workflows/nightly.yaml as workflow: input does not match expected validation schema INFO audit: zizmor: \U0001f308 completed .github/workflows/build-job-reusable.yaml INFO audit: zizmor: \U0001f308 completed .github/workflows/common-builds-reusable.yaml INFO audit: zizmor: \U0001f308 completed .github/workflows/common-test-data-reusable.yaml INFO audit: zizmor: \U0001f308 completed .github/workflows/sanitizer-job-reusable.yaml warning[artipacked]: credential persistence through GitHub Actions artifacts --> .github/workflows/build-job-reusable.yaml:61:9 | 61 | - uses: actions/checkout@v5 | _________^ 62 | | with: 63 | | fetch-depth: 50 64 | | lfs: true | |___________________^ does not set persist-credentials: false | = note: audit confidence \u2192 Low = note: this finding has an auto-fix error[template-injection]: code injection via template expansion --> .github/workflows/build-job-reusable.yaml:72:18 | 72 | run: ${{ inputs.before-script }} | --- ^^^^^^^^^^^^^^^^^^^^ may expand into attacker-controllable code | | | this run block | = note: audit confidence \u2192 High = note: this finding has an auto-fix error[unpinned-uses]: unpinned action reference --> .github/workflows/build-job-reusable.yaml:61:15 | 61 | - uses: actions/checkout@v5 | ^^^^^^^^^^^^^^^^^^^ action is not pinned to a hash (required by blanket policy) | = note: audit confidence \u2192 High = note: this finding has an auto-fix error[unpinned-uses]: unpinned action reference --> .github/workflows/build-job-reusable.yaml:74:15 | 74 | - uses: actions/cache/restore@v5 | ^^^^^^^^^^^^^^^^^^^^^^^^ action is not pinned to a hash (required by blanket policy) | = note: audit confidence \u2192 High = note: this finding has an auto-fix error[unpinned-uses]: unpinned action reference --> .github/workflows/build-job-reusable.yaml:182:15 | 182 | - uses: actions/cache/save@v5 | ^^^^^^^^^^^^^^^^^^^^^ action is not pinned to a hash (required by blanket policy) | = note: audit confidence \u2192 High = note: this finding has an auto-fix error[unpinned-uses]: unpinned action reference --> .github/workflows/build-job-reusable.yaml:188:15 | 188 | uses: actions/upload-artifact@v7 | ^^^^^^^^^^^^^^^^^^^^^^^^^^ action is not pinned to a hash (required by blanket policy) | = note: audit confidence \u2192 High = note: this finding has an auto-fix error[unpinned-images]: unpinned image references --> .github/workflows/build-job-reusable.yaml:51:18 | 51 | image: ${{ inputs.container-image }} | ^^^^^^^^^^^^^^^^^^^^^^ container image may be unpinned | = note: audit confidence \u2192 Low warning[artipacked]: credential persistence through GitHub Actions artifacts --> .github/workflows/common-builds-reusable.yaml:305:9 | 305 | - uses: actions/checkout@v5 | _________^ 306 | | with: 307 | | fetch-depth: 50 308 | | lfs: true | |___________________^ does not set persist-credentials: false | = note: audit confidence \u2192 Low = note: this finding has an auto-fix warning[artipacked]: credential persistence through GitHub Actions artifacts --> .github/workflows/common-builds-reusable.yaml:356:9 | 356 | - uses: actions/checkout@v5 | _________^ 357 | | with: 358 | | fetch-depth: 50 359 | | lfs: true | |___________________^ does not set persist-credentials: false | = note: audit confidence \u2192 Low = note: this finding has an auto-fix warning[artipacked]: credential persistence through GitHub Actions artifacts --> .github/workflows/common-builds-reusable.yaml:468:9 | 468 | - uses: actions/checkout@v5 | _________^ 469 | | with: 470 | | fetch-depth: 50 471 | | fetch-tags: true 472 | | lfs: true | |___________________^ does not set persist-credentials: false | = note: audit confidence \u2192 Low = note: this finding has an auto-fix warning[excessive-permissions]: overly broad permissions --> .github/workflows/common-builds-reusable.yaml:23:3 | 23 | / build-generic-gnu: 24 | | name: Build (generic-gnu) 25 | | uses: ./.github/workflows/build-job-reusable.yaml 26 | | if: (!cancelled()) ... | 45 | | - inspection-accounting 46 | | - no-examples | | ^ | | | | |_______________________this job | default permissions used due to no permissions: block | = note: audit confidence \u2192 Medium warning[excessive-permissions]: overly broad permissions --> .github/workflows/common-builds-reusable.yaml:48:3 | 48 | / build-x86_64-linux-gcc: 49 | | name: Build (x86_64-linux-gcc) 50 | | uses: ./.github/workflows/build-job-reusable.yaml 51 | | if: (!cancelled()) ... | 72 | | - debug 73 | | - enable-12bit-profile | | ^ | | | | |________________________________this job | default permissions used due to no permissions: block | = note: audit confidence \u2192 Medium warning[excessive-permissions]: overly broad permissions --> .github/workflows/common-builds-reusable.yaml:75:3 | 75 | / build-entropy-stats: 76 | | name: Build (entropy-stats) 77 | | uses: ./.github/workflows/build-job-reusable.yaml 78 | | if: (!cancelled()) ... | 90 | | avm-build-config: 91 | | - entropy-stats | | ^ | | | | |_________________________this job | default permissions used due to no permissions: block | = note: audit confidence \u2192 Medium warning[excessive-permissions]: overly broad permissions --> .github/workflows/common-builds-reusable.yaml:93:3 | 93 | / build-bitstream-mismatch-debug: 94 | | name: Build (bitstream-mismatch-debug) 95 | | uses: ./.github/workflows/build-job-reusable.yaml 96 | | if: (!cancelled()) ... | 105 | | avm-build-config: 106 | | - bitstream-mismatch-debug | | ^ | | | | |____________________________________this job | default permissions used due to no permissions: block | = note: audit confidence \u2192 Medium warning[excessive-permissions]: overly broad permissions --> .github/workflows/common-builds-reusable.yaml:108:3 | 108 | / build-enable-12bit-profile: 109 | | name: Build (enable-12bit-profile) 110 | | uses: ./.github/workflows/build-job-reusable.yaml 111 | | if: (!cancelled()) ... | 120 | | avm-build-config: 121 | | - enable-12bit-profile | | ^ | | | | |________________________________this job | default permissions used due to no permissions: block | = note: audit confidence \u2192 Medium warning[excessive-permissions]: overly broad permissions --> .github/workflows/common-builds-reusable.yaml:123:3 | 123 | / build-x86-linux-gcc: 124 | | name: Build (x86-linux-gcc) 125 | | uses: ./.github/workflows/build-job-reusable.yaml 126 | | if: (!cancelled()) ... | 147 | | - no-examples 148 | | - debug | | ^ | | | | |_________________this job | default permissions used due to no permissions: block | = note: audit confidence \u2192 Medium warning[excessive-permissions]: overly broad permissions --> .github/workflows/common-builds-reusable.yaml:150:3 | 150 | / build-aarch64-linux-gcc: 151 | | name: Build (aarch64-linux-gcc) 152 | | uses: ./.github/workflows/build-job-reusable.yaml 153 | | if: (!cancelled()) ... | 173 | | - inspection-accounting 174 | | - no-examples | | ^ | | | | |_______________________this job | default permissions used due to no permissions: block | = note: audit confidence \u2192 Medium warning[excessive-permissions]: overly broad permissions --> .github/workflows/common-builds-reusable.yaml:176:3 | 176 | / build-armv7-linux-gcc: 177 | | name: Build (armv7-linux-gcc) 178 | | uses: ./.github/workflows/build-job-reusable.yaml 179 | | if: (!cancelled()) ... | 199 | | - inspection-accounting 200 | | - no-examples | | ^ | | | | |_______________________this job | default permissions used due to no permissions: block | = note: audit confidence \u2192 Medium warning[excessive-permissions]: overly broad permissions --> .github/workflows/common-builds-reusable.yaml:202:3 | 202 | / build-ppc-linux-gcc: 203 | | name: Build (ppc-linux-gcc) 204 | | uses: ./.github/workflows/build-job-reusable.yaml 205 | | if: (!cancelled()) ... | 226 | | - no-examples 227 | | - debug | | ^ | | | | |_________________this job | default permissions used due to no permissions: block | = note: audit confidence \u2192 Medium warning[excessive-permissions]: overly broad permissions --> .github/workflows/common-builds-reusable.yaml:229:3 | 229 | / build-x86-mingw-gcc: 230 | | name: Build (x86-mingw-gcc) 231 | | uses: ./.github/workflows/build-job-reusable.yaml 232 | | if: (!cancelled()) ... | 251 | | - nasm 252 | | - no-examples | | ^ | | | | |_______________________this job | default permissions used due to no permissions: block | = note: audit confidence \u2192 Medium warning[excessive-permissions]: overly broad permissions --> .github/workflows/common-builds-reusable.yaml:254:3 | 254 | / build-x86_64-mingw-gcc: 255 | | name: Build (x86_64-mingw-gcc) 256 | | uses: ./.github/workflows/build-job-reusable.yaml 257 | | if: (!cancelled()) ... | 276 | | - nasm 277 | | - no-examples | | ^ | | | | |_______________________this job | default permissions used due to no permissions: block | = note: audit confidence \u2192 Medium warning[excessive-permissions]: overly broad permissions --> .github/workflows/common-builds-reusable.yaml:279:3 | 279 | / build-x86_64-clang: 280 | | name: Build (x86_64-clang) 281 | | uses: ./.github/workflows/build-job-reusable.yaml 282 | | if: (!cancelled()) ... | 294 | | - debug 295 | | - release | | ^ | | | | |___________________this job | default permissions used due to no permissions: block | = note: audit confidence \u2192 Medium error[unpinned-uses]: unpinned action reference --> .github/workflows/common-builds-reusable.yaml:305:15 | 305 | - uses: actions/checkout@v5 | ^^^^^^^^^^^^^^^^^^^ action is not pinned to a hash (required by blanket policy) | = note: audit confidence \u2192 High = note: this finding has an auto-fix error[unpinned-uses]: unpinned action reference --> .github/workflows/common-builds-reusable.yaml:329:15 | 329 | uses: actions/upload-artifact@v7 | ^^^^^^^^^^^^^^^^^^^^^^^^^^ action is not pinned to a hash (required by blanket policy) | = note: audit confidence \u2192 High = note: this finding has an auto-fix error[unpinned-uses]: unpinned action reference --> .github/workflows/common-builds-reusable.yaml:356:15 | 356 | - uses: actions/checkout@v5 | ^^^^^^^^^^^^^^^^^^^ action is not pinned to a hash (required by blanket policy) | = note: audit confidence \u2192 High = note: this finding has an auto-fix error[unpinned-uses]: unpinned action reference --> .github/workflows/common-builds-reusable.yaml:363:15 | 363 | - uses: actions/cache/restore@v5 | ^^^^^^^^^^^^^^^^^^^^^^^^ action is not pinned to a hash (required by blanket policy) | = note: audit confidence \u2192 High = note: this finding has an auto-fix error[unpinned-uses]: unpinned action reference --> .github/workflows/common-builds-reusable.yaml:430:15 | 430 | - uses: actions/cache/save@v5 | ^^^^^^^^^^^^^^^^^^^^^ action is not pinned to a hash (required by blanket policy) | = note: audit confidence \u2192 High = note: this finding has an auto-fix error[unpinned-uses]: unpinned action reference --> .github/workflows/common-builds-reusable.yaml:436:15 | 436 | uses: actions/upload-artifact@v7 | ^^^^^^^^^^^^^^^^^^^^^^^^^^ action is not pinned to a hash (required by blanket policy) | = note: audit confidence \u2192 High = note: this finding has an auto-fix error[unpinned-uses]: unpinned action reference --> .github/workflows/common-builds-reusable.yaml:468:15 | 468 | - uses: actions/checkout@v5 | ^^^^^^^^^^^^^^^^^^^ action is not pinned to a hash (required by blanket policy) | = note: audit confidence \u2192 High = note: this finding has an auto-fix error[unpinned-uses]: unpinned action reference --> .github/workflows/common-builds-reusable.yaml:476:15 | 476 | - uses: actions/cache/restore@v5 | ^^^^^^^^^^^^^^^^^^^^^^^^ action is not pinned to a hash (required by blanket policy) | = note: audit confidence \u2192 High = note: this finding has an auto-fix error[unpinned-uses]: unpinned action reference --> .github/workflows/common-builds-reusable.yaml:505:15 | 505 | - uses: actions/cache/save@v5 | ^^^^^^^^^^^^^^^^^^^^^ action is not pinned to a hash (required by blanket policy) | = note: audit confidence \u2192 High = note: this finding has an auto-fix error[unpinned-uses]: unpinned action reference --> .github/workflows/common-builds-reusable.yaml:511:15 | 511 | uses: actions/upload-artifact@v7 | ^^^^^^^^^^^^^^^^^^^^^^^^^^ action is not pinned to a hash (required by blanket policy) | = note: audit confidence \u2192 High = note: this finding has an auto-fix error[unpinned-images]: unpinned image references --> .github/workflows/common-builds-reusable.yaml:303:18 | 303 | image: ${{ inputs.container-image }} | ^^^^^^^^^^^^^^^^^^^^^^ container image may be unpinned | = note: audit confidence \u2192 Low error[unpinned-images]: unpinned image references --> .github/workflows/common-builds-reusable.yaml:342:18 | 342 | image: ${{ inputs.container-image }} | ^^^^^^^^^^^^^^^^^^^^^^ container image may be unpinned | = note: audit confidence \u2192 Low error[unpinned-images]: unpinned image references --> .github/workflows/common-builds-reusable.yaml:462:18 | 462 | image: ${{ inputs.container-image }} | ^^^^^^^^^^^^^^^^^^^^^^ container image may be unpinned | = note: audit confidence \u2192 Low warning[artipacked]: credential persistence through GitHub Actions artifacts --> .github/workflows/common-test-data-reusable.yaml:29:9 | 29 | - uses: actions/checkout@v5 | _________^ 30 | | with: 31 | | fetch-depth: 50 32 | | lfs: true | |___________________^ does not set persist-credentials: false | = note: audit confidence \u2192 Low = note: this finding has an auto-fix error[unpinned-uses]: unpinned action reference --> .github/workflows/common-test-data-reusable.yaml:29:15 | 29 | - uses: actions/checkout@v5 | ^^^^^^^^^^^^^^^^^^^ action is not pinned to a hash (required by blanket policy) | = note: audit confidence \u2192 High = note: this finding has an auto-fix error[unpinned-uses]: unpinned action reference --> .github/workflows/common-test-data-reusable.yaml:42:15 | 42 | uses: actions/upload-artifact@v7 | ^^^^^^^^^^^^^^^^^^^^^^^^^^ action is not pinned to a hash (required by blanket policy) | = note: audit confidence \u2192 High = note: this finding has an auto-fix error[unpinned-images]: unpinned image references --> .github/workflows/common-test-data-reusable.yaml:25:18 | 25 | image: ${{ inputs.container-image }} | ^^^^^^^^^^^^^^^^^^^^^^ container image may be unpinned | = note: audit confidence \u2192 Low warning[artipacked]: credential persistence through GitHub Actions artifacts --> .github/workflows/sanitizer-job-reusable.yaml:64:9 | 64 | - uses: actions/checkout@v5 | _________^ 65 | | with: 66 | | fetch-depth: 50 67 | | lfs: true | |___________________^ does not set persist-credentials: false | = note: audit confidence \u2192 Low = note: this finding has an auto-fix error[template-injection]: code injection via template expansion --> .github/workflows/sanitizer-job-reusable.yaml:98:18 | 98 | run: ${{ inputs.before-script }} | --- ^^^^^^^^^^^^^^^^^^^^ may expand into attacker-controllable code | | | this run block | = note: audit confidence \u2192 High = note: this finding has an auto-fix error[unpinned-uses]: unpinned action reference --> .github/workflows/sanitizer-job-reusable.yaml:64:15 | 64 | - uses: actions/checkout@v5 | ^^^^^^^^^^^^^^^^^^^ action is not pinned to a hash (required by blanket policy) | = note: audit confidence \u2192 High = note: this finding has an auto-fix error[unpinned-uses]: unpinned action reference --> .github/workflows/sanitizer-job-reusable.yaml:75:15 | 75 | uses: actions/download-artifact@v8 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ action is not pinned to a hash (required by blanket policy) | = note: audit confidence \u2192 High = note: this finding has an auto-fix error[unpinned-uses]: unpinned action reference --> .github/workflows/sanitizer-job-reusable.yaml:80:15 | 80 | uses: actions/download-artifact@v8 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ action is not pinned to a hash (required by blanket policy) | = note: audit confidence \u2192 High = note: this finding has an auto-fix error[unpinned-uses]: unpinned action reference --> .github/workflows/sanitizer-job-reusable.yaml:230:15 | 230 | uses: actions/upload-artifact@v7 | ^^^^^^^^^^^^^^^^^^^^^^^^^^ action is not pinned to a hash (required by blanket policy) | = note: audit confidence \u2192 High = note: this finding has an auto-fix error[unpinned-images]: unpinned image references --> .github/workflows/sanitizer-job-reusable.yaml:51:18 | 51 | image: ${{ inputs.container-image }} | ^^^^^^^^^^^^^^^^^^^^^^ container image may be unpinned | = note: audit confidence \u2192 Low 60 findings (14 suppressed, 28 unsafe fixes): 0 informational, 0 low, 18 medium, 28 high Fix Summary Successfully applied fixes to 4 files: .github/workflows/common-builds-reusable.yaml: 13 fixes .github/workflows/sanitizer-job-reusable.yaml: 6 fixes .github/workflows/build-job-reusable.yaml: 6 fixes .github/workflows/common-test-data-reusable.yaml: 3 fixes ``` </details>
…OMediaCodec#5083) Initialize `ref_costs_single` for INTRA and TIP frame indices to `INT_MAX` when they are not allowed. Previously, these elements were left uninitialized in `estimate_ref_frame_costs` if the respective modes were not allowed, but they were still being read and passed to other functions in `av2_rd_pick_inter_mode_sb`, triggering MSAN warnings. Fixes AOMediaCodec#5081
Adds Section 6.9.1 conformance check. Fixes AOMediaCodec#5085
The default value for this feature is 1, fix the comment.
prune_split_with_ml/prune_none_with_ml should be reset to 0. Set prune_split_ml_level/prune_split_ml_level_inter to - 1.
When the encoder is run in error-resilient mode (`--global-error-resilient=1`), switch frames are emitted without their reference frames being marked as restricted. This produces a bitstream that is not AV2-compliant and that decoders reject. This MR sets `restricted_prediction_switch` for error-resilient switch frames as well, so switch frames generated in that mode carry the correct restricted-reference semantics. In error-resilient mode (`--global-error-resilient=1`), the current AVM encoder converts alt-ref–type frames (which are reordered/future frames) into switch frames. When such a switch is left unconstrained (non-restricted), the encoder can assign it a low display order hint (e.g. 1) while a frame with a higher display order hint (e.g. 4) has already been output, producing a non-monotonic output order. Because a non-restricted switch does not reset the OrderHint counter, this violates the conformance constraint below (AV2 spec §7.3.6, "Coded extended layer unit"): `` If monotonic_output_order_flag is equal to 0, it is a requirement of bitstream conformance that within a coded video sequence, for a given value of obu_xlayer_id and obu_mlayer_id, if a coded output frame unit X has an associated OrderHint value equal to ohX, there shall not be a coded output frame unit Y in the same extended layer and embedded layer that appears later than X in output order and has an associated OrderHint value less than or equal to ohX, unless a switch frame with restricted_prediction_switch equal to 1 appears between X and Y in coding order. `` `` NOTE: The value of OrderHint is reset at the start of a new coded video sequence and at a switch frame with restricted_prediction_switch equal to 1. In both cases, the OrderHint counter is effectively restarted, allowing OrderHint values to be reused in subsequent coded output frame units. ``
…iaCodec#5097) The `ref_frame_sign_bias` array in the `AV2Common` structure is populated in `av2_setup_frame_sign_bias()` but never used. This patch cleans up the unused array and the associated code.
For tests with dynamic layer dropping: CDF_AVERAGING should be set to 0/off, to avoid DPB desync. Also for the MultiLayerTest2Embedded2TemporaSframe test enable_explicit_ref_frame_map_ should be set to true. Failures were seen at the lower speed settings (e.g., speed 2), not at the speed 5 setting used in the test.
In the reference commit, `cm->ref_frame_side[]` and `cm->ref_frame_relative_dist[]` were computed in two separate paths: either inside `av2_setup_motion_field()` when `allow_ref_frame_mvs` is true, or in `av2_setup_ref_frame_sides()`. In this patch, `av2_setup_ref_frame_sides()` is called unconditionally to populate these reference frame arrays, thus removing the duplicate computations inside `av2_setup_motion_field()`.
(1) Add checkout step + common setup. This resolves the issue where the 'Compare hashes' step fails with 'fatal: not a git repository' because the git command was run without checking out the repository. (2) Use `github.event.pull_request.head.sha` to get base commit SHA instead of `github.sha` which is the SHA for the "temporary merge commit". This ensures that the `git log` command to find commits containing `STATS_CHANGED` keyword works correctly. Fixes AOMediaCodec#5106
In the av2_check_profile_interop_conformance() function, the local variable `profile` is of the BITSTREAM_PROFILE type, so it is redundant to cast `profile` to BITSTREAM_PROFILE. I suspect the intended cast is `int`, to match the %d printf format specifier. Alternatively, we can simply remove the BITSTREAM_PROFILE casts.
…diaCodec#5068) (AOMediaCodec#5069) This fixes app-only builds where ENABLE_APPS=ON but ENABLE_TESTS, ENABLE_EXAMPLES, and ENABLE_TOOLS are OFF. avmdec/avmenc still need libyuv/webm include paths and object sources. Verified locally: ``` cmake --build build_fast --target avmdec -j8 cmake --build build_fast --target avmenc -j8 ``` Fixes AOMediaCodec#5068
Update enable_explicit_ref_frame_map_ setting, and remove duplicate tests (since we should set enable_explicit_ref_frame_map_ = true for these tests). Failures were seen at the lower speed settings (e.g., speed 2), not at the speed 5 setting used in the test.
As extended sdp is allowed only for partition type less than uneven 4 way search partition, this change has no effect on stats. This is done for consistency.
This change unifies uneven 4way partition search functions and h partition search functions into one function No stats changed.
- Search across ext partitions is rolled into a loop No stats changed
- Moved pruning using ml results logic to a function - Removed duplicate/unused initializations - Handled a case where none partition search gets skipped if search_none_after_rect is false and search_none_after_split is true and search none is delayed to after rect based on split results - Moved terminate partition search after none search into none search function so that it gets called correctly after none evaluation - Restored rdmult original in an early return case
The code under CONFIG_COLLECT_PARTITION_DATA is stale. This commit restores its functionality.
Check the requirement of bitstream conformance that SeenFrameHeader is not equal to is_first_tile_group. If this requirement is not met, the decoder may skip the read_uncompressed_header() call in av2_read_tilegroup_header(), which eventually leads to a null pointer dereference. Fixes AOMediaCodec#5142.
The void cast is used to ignore the tp_orig variable in Release builds because tp_orig is only used in an assert(). So the void cast should be applied to the variable itself rather than the expression *tp_orig. This is a port from libaom: fd74e18c14 Add the void cast to the tp_orig variable itself Co-authored-by: Wan-Teh Chang <wtc@google.com>
avmdec and avmenc are no longer added to AVM_EXAMPLE_TARGETS, so we
don't need to filter out avmdec and avmenc in ${AVM_EXAMPLE_TARGETS}.
The read_tile_syntax_info() and read_sequence_tile_info() functions are defined and only used in av2/decoder/decodeframe.c.
av2_highbd_warp_affine and av2_ext_highbd_warp_affine are called frequently during inter prediction at encoder speed 1. Adds NEON intrinsics for both functions, achieving 3-5x speedup over C at the kernel level. Micro-benchmark results (Apple M2 Ultra): | Kernel | C (ns) | NEON (ns) | Speedup | |--------|--------|-----------|---------| | warp 32x32 | 9209 | 1741 | 5.3x | | warp 16x16 | 2323 | 483 | 4.8x | | warp 8x8 | 589 | 153 | 3.8x | | ext_warp 16x16 | 1851 | 441 | 4.2x | | ext_warp 8x8 | 453 | 158 | 2.9x | CTC Results (RA, cpu-used=1, 33 frames, classes A4+A5): | Metric | Delta | |--------|-------| | Encode time | -5.5% | | BD-rate Y | 0.000% | | BD-rate Cb | 0.000% | | BD-rate Cr | 0.000% | Unit tests updated.
Some av2_get_chroma_format_idc() calls in av2/common/level.c and av2/common/timing.c can be omitted because the callers don't use the output argument chroma_format_idc. After these unneeded av2_get_chroma_format_idc() calls are removed, the subsampling_x, subsampling_y, monochrome parameters of some functions become unused and need to be removed.
In AV2, monochrome requires subsampling_x == 1 && subsampling_y == 1. (This somewhat arbitrary constraint comes from AV1.) Also check the return value of av2_get_chroma_format_idc() at a few call sites.
This PR enables the 12-bit profile by default and brings its implementation into conformance with AV2 Spec Draft with 12-bit support: AOMediaCodec/av2-spec-internal#776. 1. Enable CONFIG_TESTONLY_12BIT_SUPPORT by default - Flips the 12-bit config default from 0 to 1 so 12-bit encode/decode is built in by default. (Superseded by the rename below; the flag is now CONFIG_12BIT_PROFILE.) 2. Align MAIN_4xx_12_IP2 profile with AV2 spec Annex A - Renames CONFIG_TESTONLY_12BIT_SUPPORT to CONFIG_12BIT_PROFILE, and the profile enum TEST_ONLY_12BIT_PROFILE to Main_444C_12_IP2 (spec Main_444C_12_IP2, seq_profile_idc = 5). - Accepts 8/10/12-bit and all chroma formats (4:0:0/4:2:0/4:2:2/4:4:4) for profile 5, per Table A.1 (previously 8/10-bit were rejected). - Sets the max embedded-layer count for profile 5 to 3, matching its IOP2 limit (Table A.3). - Gives profile 5 its own PicSize/Bitrate factor row (36 / 3.0) per Table A.2. - Updates the in-code Annex A profile/config tables (A.1/A.2/A.5/A.6) and comments, including the bit_depth_idc mapping.
Refactor av2_rd_pick_intra_sby_mode() to evaluate Forward Skip Coding (FSC) modes using an outer fsc_mode loop, eliminating redundant candidate search logic and unifying non-FSC and FSC intra mode evaluation.
This patch refactors and abstracts different pruning logics within the intra mode evaluation loop into a new function and also avoids redundant computations in intra mode loop. This is a bit-exact change.
Abstracted code related to multi winner mode processing into a helper function `refine_winner_intra_mode_tx`. Also replaces memcpy with `av2_copy_array()`.
This patch renames variables in av2_rd_pick_intra_sby_mode() for better readablity and code consistency.
- Updated comments to adhere to coding guidelines. - Used local variables to avoid repetitive access from data structures. - Modified code logic for better readability.
Abstracted code related to luma intra mode cost calculation into a helper function to improve readability.
Coding stats shall remain the same.
The speed feature wienerns_refine_iters was in speed 2. Enabling it in speed 1 shows speedup with minor quality loss. Testset PSNR-YUV EncTime A1 (17 frames) 0.02% 95.58% A2 (33 frames) 0.03% 99.08% STATS_CHANGED Change-Id: I91fd3184f45190347c19b49eaf7bf1934238e5ea
… speed 4 (AOMediaCodec#5294) Skip unit size search for Loop Restoration based on pyramid level for speed >= 4. Anchor: e96e164 Tests: RA, A1 17 frames, others 33 frames cpu-used=4: +----------+--------+-------+--------+--------+-------------+----------+ | Summary | Y | U | V | YUV | Enc-time | Dec-time | +----------+--------+-------+--------+--------+-------------+----------+ | A1 | 0.00% | -0.02% | 0.12% | 0.00% | 95.621% | 99% | | A2 | 0.00% | -0.11% | 0.02% | 0.00% | 97.579% | 100% | | A3 | -0.03% | 0.02% | 0.02% | -0.03% | 97.019% | 100% | | A4 | 0.03% | 0.10% | -0.04% | 0.03% | 97.039% | 100% | | A5 | -0.04% | 0.20% | -0.09% | -0.03% | 97.165% | 99% | +----------+--------+--------+--------+--------+------------+----------+
Add a new speed feature to analyze the stationarity of signal to prune
partition type searches.
The speed feature shows a good tradeoff of quality and speed up.
On speed 1, RA:
Enc_Speedup PSNR-YUV Ratio
A1 (17frames) +4.75% +0.13% 36.5
A2 (33 frames) +7.39% +0.13% 56.8
STATS_CHANGED
- Consolidated initializations and moved early gating to the function start - Introduced local variables for readability - Renamed calc_masked_type_cost() to calc_comp_type_cost() and abstracted compound type cost calculation - Replaced update_mask_best_mv() with av2_copy_array() - Renamed rs2 to comp_type_rate - Simplified the masked_compound_type_rd() signature - Made cosmetic changes to follow coding guidelines No Stats Changed
- Added CompTypeRdStats to consolidate RD cost members in COMP_RD_STATS - Replaced local arrays in av2_compound_type_rd() with CompTypeRdStats This improves readability, simplifies the function signature and data copy in various helper functions. No Stats Changed
- Unified pick_wedge() and pick_wedge_fixed_sign() to avoid code duplication - Cleaned up the dead code and debug prints in pick_wedge() - Modified get_wedge_cost() to cleanup unused parameter No Stats Changed
- Modified comments to ensure correctness and correct punctuations - Replaced INLINE with AVM_INLINE - Replaced hard coded values with enums - Used pre-increments in loop counters No Stats Changed
Add default extra and encoder configuration options for
AVM_USAGE_REALTIME in av2/av2_cx_iface.c to match standard RTC encoding
defaults, and simplify test/rtc_test.cc to use the new defaults.
Setting `--rt` in the `avmenc` command line applies standard Realtime
(RTC) defaults equivalent to:
--bit-depth=8 --passes=1 --lag-in-frames=0 --auto-alt-ref=0
--enable-tpl-model=0 \
--enable-keyframe-filtering=0 --enable-deblocking=1
--enable-restoration=0 \
--enable-intra-edge-filter=0 --enable-flip-idtx=1 --enable-masked-comp=0
\
--enable-onesided-comp=0 --enable-interintra-comp=0
--enable-smooth-interintra=0 \
--enable-diff-wtd-comp=0 --enable-interinter-wedge=0
--enable-interintra-wedge=0 \
--enable-global-motion=0 --enable-warped-motion=0
--enable-smooth-intra=0 \
--enable-paeth-intra=0 --enable-cfl-intra=0 --force-video-mode=1
--enable-overlay=0 \
--enable-angle-delta=0 --enable-trellis-quant=0 --enable-qm=0
--use-intra-dct-only=1 \
--coeff-cost-upd-freq=2 --mode-cost-upd-freq=2 --mv-cost-upd-freq=3
--frame-parallel=0 \
--aq-mode=0 --deltaq-mode=0 --frame-boost=0 --noise-sensitivity=0
--cdf-update-mode=1 \
--reduced-reference-set=1 --enable-ref-frame-mvs=0 --erp-pruning-level=0
\
--use-ml-erp-pruning=0 --enable-ext-partitions=0 --enable-mrls=0
--enable-pc-wiener=0 \
--enable-wiener-nonsep=0 --enable-tip=0 --enable-bawp=0 --enable-cwp=0 \
--enable-imp-msk-bld=0 --enable-ist=0 --enable-inter-ist=0
--enable-inter-ddt=0 \
--enable-cctx=0 --enable-ibp=0 --max-drl-refmvs=0 --max-drl-refbvs=0
--enable-refmvbank=0 \
--enable-opfl-refine=0 --enable-ccso=0 --enable-lf-sub-pu=0
--enable-adaptive-mvd=0 \
--enable-flex-mvres=0 --enable-joint-mvd=0 --enable-refinemv=0
--enable-mvd-sign-derive=0 \
--enable-parity-hiding=0 --enable-warp-delta=0 --enable-warp-extend=0
--tile-columns=0 \
--gf-min-pyr-height=0 --gf-max-pyr-height=0 --enable-sdp=0
--min-gf-interval=8 \
--max-gf-interval=8 --explicit-ref-frame-map=1 --enable-bru=0
--enable-ext-seg=0 \
--enable-six-param-warp-delta=0 --enable-cdef=1
--enable-cdef-on-skip-txfm=0 \
--dpb-size=8 --use-inter-dct-only=0 --enable-fsc=1 --enable-idtx-intra=1
\
--enable-palette=1 --enable-intrabc-ext=0 --enable-intrabc=1
--cpu-used=6 \
--enable-gdf=0 --max-reference-frames=3
Set tx_size_search_level = USE_FAST_RD, and add level 3 to
restrict_tx_partition_type_search to only test
TX_PARTITION_NONE and TX_PARTITION_SPLIT.
Also switch cdef_pick_method to CDEF_FAST_SEARCH_LVL3, until the
CDEF_PICK_FROM_Q model is further evaluated/improved.
Only affects realtime mode speed >= 6.
This brings back some quality with small speed loss: on average over rtc
set of vga and 720p clips, ~10% bdrate gain with ~4% encoder speed
slowdown. This is with the command line settings below with the qp range
{70 80 90 100 110 120 130 150 170}:
--passes=1 --bit-depth=8 --obu --lag-in-frames=0 --kf-min-dist=1000
--kf-max-dist=1000 --auto-alt-ref=0 --enable-tpl-model=0
--enable-keyframe-filtering=0 --enable-deblocking=1
--enable-restoration=0 --enable-intra-edge-filter=0 --enable-flip-idtx=1
--enable-masked-comp=0 --enable-onesided-comp=0
--enable-interintra-comp=0 --enable-smooth-interintra=0
--enable-diff-wtd-comp=0 --enable-interinter-wedge=0
--enable-interintra-wedge=0 --enable-global-motion=0
--enable-warped-motion=0 --enable-smooth-intra=0 --enable-paeth-intra=0
--enable-cfl-intra=0 --force-video-mode=1 --enable-overlay=0
--enable-angle-delta=0 --enable-trellis-quant=0 --enable-qm=0
--use-intra-dct-only=1 --coeff-cost-upd-freq=2 --mode-cost-upd-freq=2
--mv-cost-upd-freq=3 --frame-parallel=0 --aq-mode=0 --deltaq-mode=0
--frame-boost=0 --noise-sensitivity=0 --cdf-update-mode=1
--reduced-reference-set=1 --enable-ref-frame-mvs=0 --erp-pruning-level=0
--use-ml-erp-pruning=0 --enable-ext-partitions=0 --enable-mrls=0
--enable-pc-wiener=0 --enable-wiener-nonsep=0 --enable-tip=0
--enable-bawp=0 --enable-cwp=0 --enable-imp-msk-bld=0 --enable-ist=0
--enable-inter-ist=0 --enable-inter-ddt=0 --enable-cctx=0 --enable-ibp=0
--max-drl-refmvs=0 --max-drl-refbvs=0 --enable-refmvbank=0
--enable-opfl-refine=0 --enable-ccso=0 --enable-lf-sub-pu=0
--enable-adaptive-mvd=0 --enable-flex-mvres=0 --enable-joint-mvd=0
--enable-refinemv=0 --enable-mvd-sign-derive=0 --enable-parity-hiding=0
--enable-warp-delta=0 --enable-warp-extend=0 --tile-columns=0
--gf-min-pyr-height=0 --gf-max-pyr-height=0 --end-usage=q
--use-fixed-qp-offsets=0 --psnr --threads=1 --enable-sdp=0
--min-gf-interval=8 --max-gf-interval=8 --explicit-ref-frame-map=0
--enable-bru=0 --enable-ext-seg=0 --enable-six-param-warp-delta=0
--enable-cdef=1 --enable-cdef-on-skip-txfm=0 --dpb-size=8
--use-inter-dct-only=0 --enable-fsc=1 --enable-idtx-intra=1
--enable-palette=1 --enable-intrabc-ext=0 --enable-intrabc=0
--tune-content=0 --rt --cpu-used=6 --enable-gdf=0
--max-reference-frames=3 --max-q=qp ---min-q=qp
Promote tx stat pruning to Speed 3 The speed feature was at speed 4. Enabling the speed feature at speed 3 shows encoder speed up with acceptable coding loss. New test on top of base commit [5d628d8](AOMediaCodec@5d628d8) Speed 3 Testset PSNR-YUV EncSpeedUp Ratio A1 (17 frames) 0.14% 8.28% 59.1 A2 (33 frames) 0.17% 6.53% 38.4 Original test on top of base commit AOMediaCodec@fe1bfde Performance on RA speed 3 Testset PSNR-YUV EncSpeedUp A1 (17 frames) 0.15% 6.55% A2 (33 frames) 0.22% 5.68% Besides, we tested the speed feature at speed 2, it does not meet the bar for A1. Performance on RA speed 2 Testset PSNR-YUV EncSpeedUp A1 (17 frames) 0.16% 4.02% A2 (33 frames) 0.13% 4.46% STATS_CHANGED
By reducing the number of candidates, we achieve a considerable speed up with accepted coding loss. Result of reducing the count to 3 for speed 4: TestSet PSNR-YUV EncTime Ratio A1 (17 frames) 0.10% 96.70% 34.1 A2 (33 frames) 0.07% 96.81% 47.0 Result of reducing the count to 3 for speed 3: TestSet PSNR-YUV EncTime Ratio A1 (17 frames) 0.11% 96.87% 29.4 A2 (33 frames) 0.09% 96.41% 41.3 If we reduce the count to 2 The speedup/loss ratio is good for A2, but not good enough for A1. On speed 4, RA, the performance is: TestSet PSNR-YUV EncTime Ratio A1 (17 frames) 0.41% 92.76% 19.0 A2 (33 frames) 0.19% 93.44% 36.9 STATS_CHANGED
…iaCodec#5304) The wet pass starts mode search from an unbounded rd limit, so its pruning cannot bite until it finds a good candidate on its own. Start it instead from the dry pass's rd for the same block and size, plus a margin. The two passes see different contexts, so the bound can be violated. A block left with no valid mode is re-searched once with the original limit. Anchor: commit c77f169 Speed 4 (cpu-used=4): FG16 CTC (33 frames, class A1 and A2, RA) ``` +------------+-------+-------+-------+-------+------+------+ | Class | Y | Cb | Cr | wAvg | Enc% | Dec% | +------------+-------+-------+-------+-------+------+------+ | A1 | -0.30 | -0.29 | -0.38 | -0.30 | 100 | 100 | | A2 | -0.17 | 0.07 | 0.31 | -0.15 | 101 | 100 | | Avg w/o B2 | -0.21 | -0.04 | 0.11 | -0.19 | 100 | 100 | +------------+-------+-------+-------+-------+------+------+ ``` STATS_CHANGED
- Introduce the reduce_max_drl_refmvs speed feature to reduce the default DRL candidate count from 4 to 3 in higher encode speeds. - Enable frame-level DRL adaptation (allow_frame_max_drl_bits = 1) when reduce_max_drl_refmvs is active: key, boosted, and low pyramid level frames (<= 2) retain 4 candidates (max_drl_bits = 3) to retain reference quality, while leaf frames use 3 candidates (max_drl_bits = 2). - Condition newmv_drl_search_limit on reduce_max_drl_refmvs to ensure both ref_mv_idx 0 and 1 are searched before reusing results, preventing motion predictor diversity collapse. - The existing --max-drl-refmvs CLI flag can override this behavior. Evaluation based on 2abcfff with 33 frames at speed 2: ``` +---------+--------+--------+--------+--------+----------+----------+ | Summary | Y | U | V | YUV | Enc-time | Dec-time | +---------+--------+--------+--------+--------+----------+----------+ | A1 | +0.05% | -0.05% | +0.10% | +0.05% | 98.1% | 100.0% | | A2 | +0.08% | +0.27% | -0.03% | +0.08% | 97.5% | 100.5% | +---------+--------+--------+--------+--------+----------+----------+ ```
Pulled common calculations from the pixel filtering loop and conducted them beforehand. CTC tests showed no stats change.
Added a speed feature to early terminate the warp delta refinement loop if the rate-distortion cost improvement in a pass drops below 5%. Results for RA CTC, A1 17 frames, A2 33 frames, speed 1: (Anchor: 7c9dfba) ``` +-----+-------+-------+-------+-------+---------------+ |Class| Y | Cb | Cr | wAvg |EncInstCount(%)| +-----+-------+-------+-------+-------+---------------+ | A2 |-0.0004|-0.0367|-0.2123|-0.0103| 98.93 | | A1 |-0.0649| 0.1274| 0.0844|-0.0513| 98.89 | +-----+-------+-------+-------+-------+---------------+ ``` STATS_CHANGED for speed >= 1
Enable frame level bawp based on histogram. Enabled for speed 2 and above. 33 frame RA results for speed 2 and 3 are as follows: +---------+--------+-------+-------+-------+----------+----------+ | Summary | Y | U | V | YUV | Enc-time | Dec-time | +---------+--------+-------+-------+-------+----------+----------+ Speed2, Overall w/o B2 (RA) +0.00% +0.01% +0.01% +0.00% 99.51% 100.12% Speed3, Overall w/o B2 (RA) +0.00% +0.01% +0.03% +0.00% 99.45% 99.53% +---------+--------+-------+-------+-------+----------+----------+
…c#5314) The expensive pixel-domain distortion path in search_tx_type() is skipped if the lightweight transform-domain RD estimate already exceeds best_rd or ref_best_rd. This is controlled by a new speed feature flag skip_pixel_dist_calc_using_tx_dist, enabled for speed >= 1. STATS_CHANGED for speed >= 1 Test results (RA) for Speed 1 Anchor: commit 5d628d8 A1 - 17 frames A2 - 33 frames ``` +------------+-------+-------+-------+-------+-------------+ | Class | Y | Cb | Cr | YUV |EncInstCount%| +------------+-------+-------+-------+-------+-------------+ | A1 | -0.03 | 0.43 | -0.06 | 0.00 | 95.84 | | A2 | 0.04 | 0.46 | -0.15 | 0.05 | 97.35 | +------------+-------+-------+-------+-------+-------------+ ```
Contributor
Author
|
Hi @urvangjoshi, @leolzhao, and @yunqingwang1, This one is target for av2-enc branch. I've rebased the entire av2-enc branch on top of the main branch. Meanwhile, I'm running the speed 0/1/4 tests on the rebased branch to confirm nothing is broken, and the results would look as expected. Feel free to review in parallel and let me know if anything needs changing.
Thanks |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Please don't squash — we need to preserve the commit history, so rebase only.