12 bit: bit depth adjustment to pc wiener filter set selection

[kylesiefring-twoorioles]

Describe the requested feature / task

pc_wiener selects it's filter sets based on qindex. There is no adjustment down for 10/12 bit, which seems incorrect.

See:
https://av2.aomedia.org/v1.0.0/index.html#pc_wiener_filter_process

The variable qindex is set equal to base_q_idx.

The variable index is set equal to get_filter_set_index(qindex).

get_filter_set_index( base_qindex ) {
    if (base_qindex < 130) {
        return 0
    } else if (base_qindex < 190) {
        return 1
    } else if (base_qindex < 220) {
        return 2
    } else {
        return 3
    }
}

Describe how the feature will be implemented / how the task will be completed

Offset by 48 (or maybe more, I guess) from the equivalent of get_filter_set_index in avm when using 12-bit and test.

@xinzhao-apple I assigned this to you since your name is on some of 12 documents. Can you delegate this out?

[xinzhao-apple]

Hi @xinzhao-apple @kylesiefring-twoorioles Thanks for the update and the discussion.

I agree Xin that this is coding performance related issue. The current filter is probably optimized for 10 bits. It would be difficult to push a coding gain fix at this moment.

BRs, Jianle

Hi @kylesiefring-twoorioles , thanks for pointing it out. Note this also affects 10-bit, not just 12-bit. If I understand it correctly, this is coding performance related issue, and without the fix, the encoder/decoder still runs properly.

That also bears on whether retraining is needed. At 8-bit the offset is zero, so the current and corrected functions are identical; if the banks were trained mainly on mixed 8-bit and 10-bit content, the training may need to be updated to see the real coding gain impact. However, at this stage, I think such normative changes will likely receive push-back.

@jianle973 , any comment on this issue?