fix: reset AV1 coding state on every key frame

[lutyjj]

NVIDIA's Vulkan AV1 encoder silently emits undecodable key frames mid-session when rate control is active: the tile data doesn't match the emitted frame header, dav1d and ffmpeg reject the frame with a hard parse error, and every frame until the next key frame is lost with it. Only the first key frame of a session was valid, so any stream using CBR/VBR with on-demand IDRs (game streaming) broke permanently on the first IDR request - in practice a permanent greenscreen the moment the client asks for a recovery key frame (this is the "potential fix" I mentioned in hgaiser/moonshine#156). CQP was unaffected, so CQP-based examples never caught it.

The fix re-issues the coding-state RESET + rate control + quality level control command on every key frame instead of only the first, same shape as the existing first-frame setup. A key frame is a clean restart point, so in theory this is generic and spec-compliant behavior regardless of the vendor (though I tested only on Nvidia). Scoped to active rate control: with rate control disabled there's no state worth resetting, so CQP skips the extra control command entirely.

Verification - new rc_keyframes example following the rfi.rs pattern: AV1 + CBR + gop_size 10 so key frames land mid-stream, then a full-stream ffmpeg decode + PSNR check against the source. On current main it fails at 12.79 dB (the stream decodes but is corrupt from the first mid-stream key frame on - the soft failure mode a plain decode-success check would miss); with this fix it passes at 67.60 dB. Also verified end to end in a real moonshine session (4K/120 AV1): pre-fix greenscreens on the first recovery key frame, post-fix recovers cleanly.

Closes hgaiser/moonshine#155.

---

Host: NVidia 5060 Ti (driver 610.43.03), Arch

[lutyjj]

jfyi: the actual fix is ~3 lines in record.rs - the rest of the diff is the rc_keyframes example, so that the issue can be showcased/reproduced without a full streaming setup (fails on main at 12.79 dB, passes with the fix at 67.60 dB).

[lutyjj]

Putting into draft for now. For some reason started observing weird behaviour on AC4 and AV1 + HDR path again. There are no green screen artifacts anymore, but on transitioning from SDR to HDR after intro movies the game now exits fullscreen (on moonlight-qt) and just black-screens. Sound goes, stream is live, but client is unable to decode frames for whatever reason. Wasn't happening few days ago on moonshine-git, need to do a little more debugging there to claim this is the definitive fix.

[lutyjj]

un-drafting - the black screen wasn't this fix. did a proper bitstream capture this time (dumped the encoded stream host-side at the SDR→HDR transition):

• stock v0.8.1: the 5 key frames after set_color_description (session params rebuild + IDR) are each invalid in isolation - dav1d refuses them outright (Error decoding frame: Invalid argument, no data decoded), frame-header region near-all-zero. The 6th key frame ~2.2s later decodes fine. That window is exactly the green screen + the client's IDR request loop. Sequence headers parse perfectly btw - it's purely the driver-emitted frame data.
• v0.8.1 + this fix: single key frame at the transition, decodes 100% clean from a cold start, no IDR loop, no green screen.

turns out the black screen was my test bundle's fault, unrelated to this PR - with this fix in I can't repro it anymore, with or without HDR metadata OBU injection on top (un-gating that injection is a separate fix, hgaiser/moonshine#166).

rebased on main (v0.8.1), rc_keyframes still passes at 67.60 dB (RTX 5060 Ti, driver 610.43.03).

[DatCaptainHorse]

You can never hate NVIDIA enough 😅

Changes look good from quick read, unfortunately got no RTX 2060 anymore to test with, since it was mostly sitting idle draining power, I sold it on the used-market. I'll verify with the RX 9060 XT when I get the chance 👍