Fix High Profile scaling matrix defaults per H.264 Table 7-2

[intrepidsilence]

Summary

When seq_scaling_matrix_present_flag is 0 in a High Profile stream, all weightScale matrices retain their flat-16 values from the struct initializer. H.264 Table 7-2 Fall-Back Rule Set A requires specific default matrices that differ from flat-16.

The Bug

The struct initializer at line ~1826 sets all scaling matrices to flat-16:

.weightScale4x4_v = {[0 ... 5] = {16, 16, ...}},
.weightScale8x8_v = {[0 ... 23] = {16, 16, ...}},

The existing code at line ~1869 only sets defaults for 4x4 lists 0 and 3 (Intra/Inter), and only inside the seq_scaling_matrix_present_flag == 1 block. This means:

• When flag == 0: all lists stay at flat-16 (wrong)
• When flag == 1: lists 1, 2, 4, 5 use flat-16 as fall-back instead of spec defaults (wrong)

Fix

Set all spec-mandated defaults before checking seq_scaling_matrix_present_flag:

// H.264 Table 7-2 Fall-Back Rule Set A
sps.weightScale4x4_v[0] = Default_4x4_Intra;  // lists 0-2 cascade
sps.weightScale4x4_v[1] = Default_4x4_Intra;
sps.weightScale4x4_v[2] = Default_4x4_Intra;
sps.weightScale4x4_v[3] = Default_4x4_Inter;  // lists 3-5 cascade
sps.weightScale4x4_v[4] = Default_4x4_Inter;
sps.weightScale4x4_v[5] = Default_4x4_Inter;
for (int i = 0; i < 4; i++) {
    sps.weightScale8x8_v[i] = Default_8x8_Intra[i];
    sps.weightScale8x8_v[4 + i] = Default_8x8_Inter[i];
}
if (get_u1(&dec->gb)) { // seq_scaling_matrix_present_flag
    // parse_scaling_lists overrides with stream-provided values
    parse_scaling_lists(...);
}

Symptom

Inter-prediction drift that grows progressively across each GOP:

• Frame 0 (IDR): mean pixel difference vs FFmpeg = 0.26
• Frame 12 (P-frame): mean diff = 4.83
• Frame 24 (end of GOP): mean diff = 17.11
• After next IDR: resets to 0.26, then grows again

The incorrect flat-16 dequantization causes subtle errors in intra-coded macroblocks that propagate and accumulate through inter-prediction.

Verification

• Tested on commercial 3D Blu-ray MVC streams (High Profile, seq_scaling_matrix_present_flag=0)
• Before fix: mean absolute difference vs FFmpeg grows from 0.26 to 17.11 per GOP
• After fix: pixel-perfect match with FFmpeg (0.0 mean diff for 500+ consecutive frames)
• 85/85 JVT conformance streams continue to pass

[traffaillac]

Hi!
Thanks for taking the time to submit PRs, and thanks for making them so detailed! I'll need some time to review them, also can you send me a PM with sample bitstreams to observe these problems? It is especially important to ensure that I won't break their decoding in the future (I will have to rewrite the DPB code for PAFF/MBAFF). I may also make a script soon to strip the YAML output from its image content, tell me if it may allow you to send me a sample.
Cheers,
Thibault

[intrepidsilence]

@traffaillac The file I was converting was an MKV dump of the movie "Gravity (2013)" from 3D BD. Since this is copyright protected I cannot send. I used makemkv to dump it. The MKV file played cleanly in Bino, which is the only other app I know on macOS that can read the 3D format. I was also able to programmatically confirm that the left and right images in the source were devoid of issues. I think this and the other 3 patches I submitted may bring edge 264 much closer to ready for production.

[Frankaboy7]

Hi Thibault,
I’d like to test edge264, updated with the four pull requests from intrepidsilence on my Arch Linux system, please.
I have the movie “Gravity (2013)” on 3D Blu-ray too.
If you’d like, I can send you the a test sample (sliced with Mkvtoolnix, 800Mb large)
from “Gravity (2013)” via WeTransfer.