Skip to content

Unnest compound mode search loops and clean up inter search - #5343

Draft
kslu-aom wants to merge 3 commits into
AOMediaCodec:av2-encfrom
kslu-aom:av2enc-hcip
Draft

Unnest compound mode search loops and clean up inter search#5343
kslu-aom wants to merge 3 commits into
AOMediaCodec:av2-encfrom
kslu-aom:av2enc-hcip

Conversation

@kslu-aom

@kslu-aom kslu-aom commented Aug 28, 2026

Copy link
Copy Markdown
Contributor

This change includes the following cleanups

  • Unnest JMVD scale factor, CWP, and RefineMV loops in handle_compound_inter_prediction to avoid redundant motion search and skip invalid tool combinations. The new search order is 1) base mode, 2) refine MV, 3) CWP, 4) JMVD scale. This search order is intended to allow potential early termination in the future
  • Remove unused variables, unnecessary checks, redundant mbmi field re-assignments, and move the update of common variable outside of search loops.
  • Use reset_inter_mode_info helper to unify mode_info field resets

STATS_CHANGED

Results tested with RA 33 frames based on ae462ed

    1) Speed 0 (partial)
      +------------+-------+-------+-------+-------+-------+-------+
      | Class      |     Y |    Cb |    Cr |  wAvg |  Enc% |  Dec% |
      +------------+-------+-------+-------+-------+-------+-------+
      | A1         | -0.02 | -0.03 | +0.04 | -0.01 | 100.1 |  99.3 |
      | A2         | -0.00 | -0.03 | -0.04 | -0.01 |  99.9 |  99.8 |
      +------------+-------+-------+-------+-------+-------+-------+
    
    2) Speed 1
      +------------+-------+-------+-------+-------+-------+-------+
      | Class      |     Y |    Cb |    Cr |  wAvg |  Enc% |  Dec% |
      +------------+-------+-------+-------+-------+-------+-------+
      | A1         | +0.01 | +0.02 | +0.03 | +0.01 |  99.3 |  99.0 |
      | A2         | +0.01 | +0.01 | -0.06 | +0.00 |  99.8 |  99.3 |
      +------------+-------+-------+-------+-------+-------+-------+
    
    3) Speed 2
      +------------+-------+-------+-------+-------+-------+-------+
      | Class      |     Y |    Cb |    Cr |  wAvg |  Enc% |  Dec% |
      +------------+-------+-------+-------+-------+-------+-------+
      | A1         | +0.00 | +0.00 | -0.11 | -0.01 | 100.8 |  99.7 |
      | A2         | +0.01 | +0.02 | -0.01 | +0.01 | 100.2 | 100.1 |
      +------------+-------+-------+-------+-------+-------+-------+

Unnest the JMVD scale factor, CWP, and RefineMV loops in
handle_compound_inter_prediction into sequential evaluation stages:
1. Base compound mode (scale 0, CWP EQUAL, RefineMV 0)
2. RefineMV mode (scale 0, CWP EQUAL, RefineMV 1)
3. CWP modes (scale 0, CWP != EQUAL, RefineMV 0)
4. JMVD scale factor modes (scale 1..N-1, CWP EQUAL, RefineMV 0)

Avoid redundant motion search and skip invalid tool combinations.
…prediction

- Hoist invariant mbmi initializations, rate computations, and loop invariants
  out of search loops in handle_single_inter_prediction and
  handle_compound_inter_prediction.
- Use reset_inter_mode_info helper to unify mode_info field resets.
- Use base template struct assignment in compound search instead of
  re-assigning individual invariant fields per iteration.
- Clean up BAWP search in single prediction by simplifying scalar BAWP MV
  tracking and mode_info access.
- Remove redundant dead checks in CWP and JMVD scaling stages.
@kslu-aom

Copy link
Copy Markdown
Contributor Author

Please review this refactoring patch and let me know if anything can be improved. I have a follow up patch to clean up more variables, search state structs, and function signatures, but I think it is better to separate this unnesting change for code review. I will mark the PR as ready when speed 0 tests are finished.

@yeqing-wu yeqing-wu left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kslu-aom, thanks very much for helping refactor these code, it looks good to me. Just have one question that need your confirm.

Comment thread av2/encoder/rdopt.c
&it_ctx, bsize, ref_mv_idx[0], ref_mv_idx[1], precision_dx, bawp_flag,
ref_mv_idx_type, 0, single_cwp_mask, this_mode, refs, flex_mv_cost,
drl_cost, jmvd_scale_mode_cost, base_rate, cur_mv, rate_mv,
ref_mv_idx_type, 0, NULL, this_mode, refs, flex_mv_cost,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The old code allocated int single_cwp_mask[MAX_CWP_NUM] = { 0 }; single_cwp_mask[0] = 1; on the stack and passed that. The new code passes NULL.

Would this cause a problem? Please check.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a minor cleanup for handle_single_inter_prediction. Since CWP and JMVD are irrelevant to the single prediction case, the mask and cost are removed from here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants