Reuse one scaler destination frame per decode thread

[nnmarcoo]

One allocation per decoded frame, and the FFmpeg bump needed to build it

Two independent commits. The FFmpeg move stands on its own and fixes a release job that was already broken. The allocation fix is the actual optimization, and it only pays off above roughly 2.5K.

frame_to_image allocated a full frame per call

frame_to_image built a fresh ffmpeg::frame::Video::empty() every time it ran. ffmpeg-next's Context::run allocates whenever the destination frame is empty:

if output.is_empty() {
    output.alloc(self.output.format, self.output.width, self.output.height);
}

So every decoded frame called av_frame_get_buffer on the way in and freed the buffer when the local dropped. At 4K that is 33 MB allocated and released per frame, before the copy into ImageData even starts.

The odd part is that the decode loop already keeps a Scaler and a decoded frame alive across iterations. The destination frame was the only one being rebuilt. It now sits beside them in run_decode, which is why build_frame and frame_to_image each take one more argument.

The copy that stays

to_vec is still there and is now load-bearing. Reusing the scaler buffer means the data has to be copied out before the next call overwrites it. Dropping that copy too would need a buffer pool with a return path from the consumer back to the decode thread, since ImageData owns its Vec<u8> and outlives the frame. That is a much bigger change and I have no evidence yet that it is worth it.

Measured

A/B in one process, release profile, timing only the scale and copy step so decode cost does not drown the signal. Seven reps with alternating order, run twice.

resolution	bytes per frame	run 1	run 2
1920x1080	8.3 MB	+0.5%	-0.1%
2880x1920	22 MB	-13.1%	-12.7%
3840x2160	33 MB	-11.1%	-10.7%

1080p gains nothing, and that result is repeatable. glibc raises its mmap threshold when a large block is freed, so an 8.3 MB buffer gets recycled from the heap and the allocation I removed was already close to free. Past about 22 MB the recycling stops and every frame pays mmap, page faults, and munmap.

So the claim is narrow: 10 to 13% off the per-frame conversion cost at 2.5K and above, nothing at 1080p, no regression measured anywhere.

The audio path is left alone deliberately

The resampler call has the same shape and I am not touching it. swresample sizes its allocation to input.samples(), so a reused frame caps at the first packet's sample count, and later samples get buffered inside swresample instead. That would perturb the out.samples() arithmetic and the tail at EOF. The prize is about 8 KB a frame against 33 MB on the video side.

FFmpeg 9

ffmpeg-next 8 cannot compile against FFmpeg 9 headers. The generated bindings drop symbols the crate matches on exhaustively, so side_data.rs fails E0004 on AV_PKT_DATA_HEVC_CONF and codec/id.rs fails E0425 on AV_CODEC_ID_V410, before any of our code compiles. On a current distro the av feature simply cannot be built. Arch is already on n9.0.1.

macOS release was broken by this and nothing caught it. That job runs an unpinned brew install ffmpeg, and brew's stable is 9.0.1. Linux and Windows kept passing because they pin an n8.1 build, so CI stayed green while one third of the matrix could not have worked.

ffmpeg-next 9 needs zero source changes here, so the commit is a version bump plus the pinned downloads that have to move with it: the BtbN asset regexes and error strings in ci.yml and release.yml, the win64 URL in setup-av.ps1, and the extracted-directory filters matching on the ffmpeg-n8.1 prefix.

Testing

437 tests pass with --features heif,av, GPU tests included. Clippy is clean at -D warnings. The release binary builds and links against system libavcodec 63. Both n9.0 assets resolve through the same API query CI runs, and the hardcoded PowerShell URL returns 200.

None of that touches the decode path, which has no test coverage at all. The change needs a real video played and scrubbed before I would trust it.

Known gap, not fixed here

setup-av.sh on Linux only checks that libavformat exists, with no version constraint. It prints "Done. Run: cargo build --features av" on a machine where the build cannot possibly work, which is exactly how this started. A pkg-config --atleast-version guard would turn it into a real check.