Skip to content

fix: submit packed headers required by the VA-API driver - #26

Open
maryny4 wants to merge 1 commit into
MuNeNiCK:mainfrom
maryny4:fix/vaapi-packed-headers
Open

fix: submit packed headers required by the VA-API driver#26
maryny4 wants to merge 1 commit into
MuNeNiCK:mainfrom
maryny4:fix/vaapi-packed-headers

Conversation

@maryny4

@maryny4 maryny4 commented Aug 6, 2026

Copy link
Copy Markdown

Root cause of #21

Mesa radeonsi requires the application to submit packed headers (VAConfigAttribEncPackedHeaders = SEQUENCE | PICTURE | SLICE | …). vlVa parses the packed slice header to learn nal_ref_idc / nal_unit_type and the slice-header fields the firmware writes into the final slice NAL (picture_h264_enc.c: parseEncSliceParamsH264), and composes SPS/PPS itself from the packed sequence data (radeon_vcn_enc.c: radeon_vcn_enc_encode_h264_header). Without packed headers the emitted slices carry a zeroed NAL header byte and reference parameter sets that are never inserted, so no decoder accepts a single frame: clients render a permanent white screen while input keeps working. Intel iHD composes headers itself, which is why the issue is AMD-only.

Offline reproduction on radeonsi (Raphael iGPU, Mesa 26.1.6), no RDP client needed — the new #[ignore]d vaapi_encode_probe test encodes solid-color frames and dumps the Annex-B stream:

  • before: 30 frames = 2 KB total, every slice NAL is type 0 (00 00 00 01 00 88 80 40 02 …), ffmpeg decodes zero frames;
  • an ffmpeg h264_vaapi control stream on the same device is byte-for-byte structurally identical except the proper 65 NAL byte — hardware and driver are fine, the submission is what differs.

Fix

Query VAConfigAttribEncPackedHeaders at encoder creation; when the driver asks for them, submit packed sequence (SPS+PPS) buffers on IDR frames and a packed slice header per frame, built with the existing BitWriter and consistent with generate_sps_pps / build_slice_params. On radeonsi the driver then writes SPS/PPS itself (taking only the NAL byte from the packed data) and derives the slice NAL from the parsed header, so the parameter sets and slices always come from one composer. Drivers that do not request packed headers keep the exact previous behavior.

After the fix the probe stream contains proper IDR/P slices and decodes to the expected solid color, matching the ffmpeg control output.

Relation to #22

#22 diagnoses the same white screen and confirms it on VCN 3.0 — credit to @stevenwcarter for the finding that the synthesized PPS advertised syntax (transform_8x8) the firmware never emits. This PR takes the complementary contract-level route: with packed headers submitted, radeonsi composes SPS/PPS and slice NALs itself, so the stream is self-consistent without post-hoc NAL repair or vendor-string gating, and the synthesized-PPS mismatches cannot occur by construction. The encoder selection option proposed separately also gives affected users an explicit software fallback.

cargo fmt --check, clippy -- -D warnings, cargo test (both feature sets) pass.

Fixes #21

Mesa radeonsi expects the application to submit packed SPS/PPS and
slice headers (VAConfigAttribEncPackedHeaders); vlVa parses the packed
slice header for nal_ref_idc/nal_unit_type and the slice fields the
firmware writes into the final slice NAL. Without them the emitted
slices carry a zeroed NAL header byte and reference parameter sets
that are never inserted, so the stream never decodes: clients render
a permanent white screen while input keeps working (MuNeNiCK#21). Intel iHD
composes headers itself, which is why the issue was AMD-only.

Query VAConfigAttribEncPackedHeaders at encoder creation and submit
packed sequence (SPS+PPS) buffers on IDR frames plus a packed slice
header per frame when the driver asks for them.

Verified offline on radeonsi (Raphael iGPU, Mesa 26.1.6) with the
ignored vaapi_encode_probe test: previously the stream contained only
type-0 NAL blobs and zero decodable frames; with packed headers it
contains proper IDR/P slices and decodes to the expected solid color.

Fixes MuNeNiCK#21

(cherry picked from commit e57ee62)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Whitescreen during connection but session otherwise works

1 participant