Encoder limits prediction modes for restricted switch frames

[mabenjak]

What version / commit were you testing with?

Current main branch commit 636fcda

What steps will reproduce the problem?

Encode the same bitstream with restricted and again with non-restricted switch frames in random access configuration. The rate of the switch frames is expect to be close to each other in both cases. The rate of the non-restricted switch frame is however much lower than the rate of the restricted switch frame.

Additional information

The function prune_ref_by_selective_ref_frame() returns 1 for all blocks that reference restricted references:

static INLINE int prune_ref_by_selective_ref_frame(
    const AV2_COMP *const cpi, const MACROBLOCK *const x,
    const MV_REFERENCE_FRAME *const ref_frame) {
  (void)x;
  const AV2_COMMON *const cm = &cpi->common;
  const SPEED_FEATURES *const sf = &cpi->sf;

  if (!sf->inter_sf.selective_ref_frame) return 0;
  assert(ref_frame[0] != NONE_FRAME);
  if (ref_frame[0] == INTRA_FRAME) return 0;

  RefCntBuffer *ref0_buf = get_ref_frame_buf(cm, ref_frame[0]);
  if (ref0_buf)
    if (ref0_buf->is_restricted) return 1;
  RefCntBuffer *ref1_buf = get_ref_frame_buf(cm, ref_frame[1]);
  if (ref1_buf)
    if (ref1_buf->is_restricted) return 1;

This causes WARPMV to be the only prediction mode to be considered for restricted references by the encoder, severely hindering the compression performance of restricted switch frames and post-switch frames referencing restricted references.