We have identified two bugs regarding BAWP in the AV2 specification that can be triggered when SWITCH OBUs are present in a bitstream and if a switching process has occurred.
BAWP reference-sample access
It is quite possible that after switching, and if a frame contains BAWP predicted blocks, because of pixel drift motion vectors that are currently “disallowed” to be present in combination with BAWP (i.e. motion vectors pointing outside the valid image) to be encountered in the bitstream. This can result in unspecified behavior in the decoder since such a bitstream would not be conforming to the current specification.
AV2 specification §7.13.3.25: “Block adaptive weighted prediction process”:
It is a requirement of bitstream conformance that all the following are true whenever this process is invoked:
refX is greater than or equal to 1.
refY is greater than or equal to 1.
refX + bw is less than or equal to planeWidth.
refY + bh is less than or equal to planeHeight.
The ideal (however, normative) solution would be to alter this restriction and instead permit such combination. This could be done by clipping the reference sample positions to always be within the frame bounds. In particular the pseudocode in subsection 7*.13.3.25. Block adaptive weighted prediction process should be modified as follows:*
if ( plane == 0 ) {
plane = 0
subX = 0
subY = 0
} else {
subX = SubsamplingX
subY = SubsamplingY
}
planeWidth = MiCols * MI_SIZE >> subX
planeHeight = MiRows * MI_SIZE >> subY
bw = Min(planeWidth - x, w)
bh = Min(planeHeight - y, h)
dy = to_fullmv( mv[0] )
dx = to_fullmv( mv[1] )
refY = ( MiRow * MI_SIZE + dy ) >> subY
refX = ( MiCol * MI_SIZE + dx ) >> subX
refX = Clip3(1, planeWidth - bw, refX)
refY = Clip3(1, planeHeight - bh, refY)
It should be noted that bitstreams generated by the current encoder, even without this modification, still satisfy this modification since locations that would require such clipping are excluded from the mode decision and are never encountered in any of the CTC bitstreams. However, we can also modify the encoder to consider such locations if it is truly desired. To accelerate, however, the decision, we would recommend to not require this evaluation for proceeding with a CWG decision (CTC results for this proposal can be assumed to be “at least” identical with the latest CTC results).
If this solution is not possible given the current status of the specification, the alternative could be to require that if a bitstream utilizes SWITCH OBUs then it is a requirement that either no BAWP prediction is used (safe restriction; easy to verify; however there will be a performance in coding impact) or that it is guaranteed that BAWP is never associated with motion vectors that are out of bounds (requires the decoder to always validate the vectors of BAWP and for the encoder/service to validate the bitstream).
high level syntax desynchronization
Because the presence of the BAWP and motion-vector-refinement syntax elements depends on reference state, a non-restricted SWITCH frame can desynchronize high-level syntax parsing and cause severe, unrecoverable decoding failures; not merely reconstruction drift. We therefore recommend that the use of non-restricted SWITCH frames be completely discouraged, and ideally disallowed: any switch that changes the reference chain should use restricted references (restricted_prediction_switch), so that the reference-dependent tools are disabled at the switch and the parsing hazard cannot arise. Alternatively, if non-restricted SWITCH frames are to be used, no frames prior to the SWITCH frame should be retained for subsequent prediction.
We have identified two bugs regarding BAWP in the AV2 specification that can be triggered when SWITCH OBUs are present in a bitstream and if a switching process has occurred.
BAWP reference-sample access
It is quite possible that after switching, and if a frame contains BAWP predicted blocks, because of pixel drift motion vectors that are currently “disallowed” to be present in combination with BAWP (i.e. motion vectors pointing outside the valid image) to be encountered in the bitstream. This can result in unspecified behavior in the decoder since such a bitstream would not be conforming to the current specification.
AV2 specification §7.13.3.25: “Block adaptive weighted prediction process”:
The ideal (however, normative) solution would be to alter this restriction and instead permit such combination. This could be done by clipping the reference sample positions to always be within the frame bounds. In particular the pseudocode in subsection 7*.13.3.25. Block adaptive weighted prediction process should be modified as follows:*
It should be noted that bitstreams generated by the current encoder, even without this modification, still satisfy this modification since locations that would require such clipping are excluded from the mode decision and are never encountered in any of the CTC bitstreams. However, we can also modify the encoder to consider such locations if it is truly desired. To accelerate, however, the decision, we would recommend to not require this evaluation for proceeding with a CWG decision (CTC results for this proposal can be assumed to be “at least” identical with the latest CTC results).
If this solution is not possible given the current status of the specification, the alternative could be to require that if a bitstream utilizes SWITCH OBUs then it is a requirement that either no BAWP prediction is used (safe restriction; easy to verify; however there will be a performance in coding impact) or that it is guaranteed that BAWP is never associated with motion vectors that are out of bounds (requires the decoder to always validate the vectors of BAWP and for the encoder/service to validate the bitstream).
high level syntax desynchronization
Because the presence of the BAWP and motion-vector-refinement syntax elements depends on reference state, a non-restricted SWITCH frame can desynchronize high-level syntax parsing and cause severe, unrecoverable decoding failures; not merely reconstruction drift. We therefore recommend that the use of non-restricted SWITCH frames be completely discouraged, and ideally disallowed: any switch that changes the reference chain should use restricted references (restricted_prediction_switch), so that the reference-dependent tools are disabled at the switch and the parsing hazard cannot arise. Alternatively, if non-restricted SWITCH frames are to be used, no frames prior to the SWITCH frame should be retained for subsequent prediction.