H.264/HEVC decoders silently produce near-uniform gray frames for real-world MP4s; encoder→decoder roundtrip fails reconstruction (v0.1.10)

[chh-itt]

Summary

While evaluating yscv-video 0.1.10 as the decode backend for a pure-Rust GUI video player,
we found that Mp4VideoReader::next_frame() returns Ok(Some(frame)) with near-uniform
gray pixel data (mean ≈ 128, std ≈ 0–8) for every real-world MP4 we tested — including the
very first IDR frame. No error is ever surfaced. A self-contained encoder→decoder roundtrip
(no external files) also fails to reconstruct the picture.

Container parsing itself appears fine: dimensions, frame counts and sample extraction are all
correct. The failure seems isolated to pixel reconstruction, and it fails silently.

Environment

• yscv-video 0.1.10 (crates.io, default features, software decode path)
• rustc 1.95.0, Windows 11 x86_64 (AVX2)
• cargo build --release

Repro 1 — self-contained roundtrip (no downloads needed)

Encode 60 frames of a gradient background with a white square moving 8 px/frame,
feed the Annex-B output straight back into H264Decoder via parse_annex_b:

use yscv_video::{H264Decoder, H264Encoder, parse_annex_b, rgb8_to_yuv420};

fn main() {
    let (w, h) = (640usize, 352usize); // NOTE: h=360 panics, see "Bonus" below
    let mut enc = H264Encoder::new(w as u32, h as u32, 28);
    let mut dec = H264Decoder::new();
    let (mut total, mut found) = (0usize, 0usize);
    for f in 0..60usize {
        let mut rgb = vec![0u8; w * h * 3];
        for y in 0..h {
            for x in 0..w {
                let i = (y * w + x) * 3;
                rgb[i] = (x * 255 / w) as u8;
                rgb[i + 1] = (y * 255 / h) as u8;
                rgb[i + 2] = 64;
            }
        }
        let sq = 20 + f * 8;
        for y in 100..180 {
            for x in sq..(sq + 80).min(w) {
                let i = (y * w + x) * 3;
                rgb[i] = 255; rgb[i + 1] = 255; rgb[i + 2] = 255;
            }
        }
        let au = enc.encode_frame(&rgb8_to_yuv420(&rgb, w, h));
        for nal in parse_annex_b(&au) {
            if let Ok(Some(fr)) = dec.process_nal(&nal) {
                total += 1;
                let row = 140 * fr.width * 3;
                let hit = (0..fr.width).any(|x| {
                    let i = row + x * 3;
                    fr.rgb8_data[i] > 200 && fr.rgb8_data[i + 1] > 200 && fr.rgb8_data[i + 2] > 200
                });
                if hit { found += 1; }
            }
        }
    }
    println!("decoded {total}/60, white square visible in {found}/{total} frames");
}

Expected: square visible in ~60/60 frames, at monotonically increasing x.
Actual: decoded 60/60, white square visible in 2/60 frames — and per-frame mean
luminance drifts wildly (69→98) even though the source scene has constant brightness.

Repro 2 — real-world files decode to gray

use yscv_video::Mp4VideoReader;

fn main() {
    let mut r = Mp4VideoReader::open(std::path::Path::new("input.mp4")).unwrap();
    let mut idx = 0usize;
    while let Ok(Some(f)) = r.next_frame() {
        if idx % 20 == 0 {
            let n = f.rgb8_data.len() as f64;
            let mean = f.rgb8_data.iter().map(|&v| v as f64).sum::<f64>() / n;
            let var = f.rgb8_data.iter().map(|&v| (v as f64 - mean).powi(2)).sum::<f64>() / n;
            println!("frame {idx}: {}x{} mean={mean:.2} std={:.2}", f.width, f.height, var.sqrt());
        }
        idx += 1;
    }
    println!("total decoded: {idx}");
}

Results across 4 independent public files (natural video should have std ≈ 30–70):

File	Codec / profile	Frames decoded	Typical output
BBB 1080p H.264	AVC High (CABAC, B-frames)	600/600 ✓	mean=127.89 std=7.5
BBB 1080p H.265	HEVC Main	300/300 ✓	mean=128.00 std=0.1–1.2
Jellyfish 1080p H.264	AVC High	300/300 ✓	mean=128.06 std=1.8–3.0
w3schools mov_bbb.mp4	AVC Constrained Baseline (CAVLC, no B-frames)	all ✓	frame 0 (IDR): mean=128.00 std=0.00

Notes:

• The very first IDR of a Constrained Baseline stream decodes to a literally uniform
  128-gray frame (std=0.00), so the failure is already in the intra path — before any
  inter/B-frame machinery is involved. CAVLC-only content fails too, so it is not
  CABAC-specific.
• Dimensions, frame counts, bit_depth and keyframe flags are all correct, and consecutive
  frames differ by tiny amounts (MAD < 0.25), i.e. the decoder emits initialized-but-mostly-
  unreconstructed buffers rather than erroring out.
• Decode throughput is excellent (3.3 ms/frame @1080p H.264 on our machine), which makes the
  silent-garbage failure mode easy to miss in benchmarks.

Bonus bugs found along the way

1. H264Encoder::encode_frame panics (index out of bounds, h264_encoder.rs:1424) when
   height is not a multiple of 16 (e.g. 640×360).
2. When reconstruction fails, no Err/log is produced — a debug assertion or an error path
   (like the "decoder probably failed silently" check in your own hevc_integration.rs)
   promoted to the library itself would make this much easier to catch.

Question

Is real-world H.264/HEVC content (x264/x265-encoded MP4s) expected to work in 0.1.x, or is
that part of the v0.2.0 milestone tracked in #4 / #5? We'd genuinely like to adopt yscv as a
pure-Rust decode backend and are happy to provide more repro files, logs, or bisect help.

[enthropy7]

Hi, thanks for reporting the bug. Yes, yscv-video will continue to work stably in the 0.1.x branch; for 0.2.0, we’re planning a full, bug-free release, just room for improvement. I’ve been focused on the kernels and the math, and right now the inference itself (the framework’s main goal) remains a P0 priority task. I’ve started working on fixing the codecs a while ago, and I have the fixed versions locally on my PC in one of the branches. Thanks to your report, I realized there are still a couple of issues I haven’t resolved yet.

To be honest, the problem with the decoder itself isn't so much that there's a bug. There WAS an incomplete framework for a couple of important features. It actually doesn't take much work. I'll try to finish everything in the next couple of days. I'll keep you posted, and I'll provide whatever support you need for your tasks.