fix: produce decodable H.264 on radeonsi (AMD) VA-API encoders#22
Open
stevenwcarter wants to merge 1 commit into
Open
fix: produce decodable H.264 on radeonsi (AMD) VA-API encoders#22stevenwcarter wants to merge 1 commit into
stevenwcarter wants to merge 1 commit into
Conversation
On AMD GPUs the VA-API hardware path produced a stream every decoder rejects, showing a black or white screen while input, clipboard and audio kept working (MuNeNiCK#21). Two causes, both rooted in radeonsi expecting packed headers that hypr-rdp does not submit: - The slice NAL unit header byte comes back as 0x00 (invalid) because the driver only fills it from packed slice headers. Repair it from frame state; a valid stream never contains type-0 NALs, so this is a no-op on drivers that fill the byte (e.g. Intel iHD). - The synthesized PPS declared transform_8x8_mode_flag=1, but VCN hardware before 5.0 cannot emit that syntax, so decoders parsed phantom bits. Disable it, and align POC type and deblocking signaling with what the driver actually writes. The parameter changes are gated on the driver vendor string (radeonsi): all other drivers keep byte-identical legacy behavior. The NAL repair is unconditional but only rewrites invalid NALs. Verified on RX 6800 XT (VCN 3.0, Mesa 26.1.5), Hyprland 0.55.4, macOS Windows App client at 3456x2160: hardware encoding now works. Fixes MuNeNiCK#21
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
On AMD GPUs the VA-API hardware path produced a stream every decoder rejects, showing a black or white screen while input, clipboard and audio kept working (#21). Two causes, both rooted in radeonsi expecting packed headers that hypr-rdp does not submit:
The slice NAL unit header byte comes back as 0x00 (invalid) because the driver only fills it from packed slice headers. Repair it from frame state; a valid stream never contains type-0 NALs, so this is a no-op on drivers that fill the byte (e.g. Intel iHD).
The synthesized PPS declared transform_8x8_mode_flag=1, but VCN hardware before 5.0 cannot emit that syntax, so decoders parsed phantom bits. Disable it, and align POC type and deblocking signaling with what the driver actually writes.
The parameter changes are gated on the driver vendor string (radeonsi): all other drivers keep byte-identical legacy behavior. The NAL repair is unconditional but only rewrites invalid NALs.
Verified on RX 6800 XT (VCN 3.0, Mesa 26.1.5), Hyprland 0.55.4, macOS Windows App client at 3456x2160: hardware encoding now works.
Fixes #21