fix: skip HRD parameters when no target bitrate is set - #30
Open
maryny4 wants to merge 1 commit into
Open
Conversation
Mesa radeonsi segfaults inside vaEndPicture when it receives a VAEncMiscParameterHRD buffer with buffer_size = 0, which is exactly what CQP mode submitted (bits_per_second = 0): every client connection crashed the server in a restart loop. ffmpeg's h264_vaapi works on the same driver because it never sends HRD parameters without a bitrate. Bisected offline with the vaapi_encode_probe harness: rate-control and frame-rate buffers are fine, any combination including the zero-sized HRD buffer reproduces the segfault. HRD is meaningless without a target bitrate, so skip it in that case; a CQP variant of the probe now guards the path.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Selecting
rate_control = "cqp"crashes the server on every client connection (SIGSEGV in a systemd restart loop). The crashing frame is inside Mesa radeonsi viavaEndPicture.CQP mode has no target bitrate, so the rate-control policy carries
bits_per_second = 0— and the HRD misc buffer was built from it:buffer_size = 0,initial_buffer_fullness = 0. Mesa radeonsi segfaults on a zero-sized HRD buffer instead of rejecting it (driver bug, to be reported separately); ffmpeg'sh264_vaapiworks on the same driver because it never submits HRD parameters without a bitrate.Bisected offline with the
vaapi_encode_probeharness on radeonsi (Raphael iGPU, Mesa 26.1.6): the rate-control and frame-rate misc buffers are fine in every combination; any combination that includes the zero-sized HRD buffer reproduces the segfault.Fix
HRD parameters are meaningless without a target bitrate — skip the HRD buffer when
bits_per_second = 0. VBR behavior is unchanged. A CQP variant of the ignored probe test now guards the path; after the fix it produces a valid stream that decodes to the expected solid color.cargo fmt --check,clippy -- -D warnings,cargo test(both feature sets) pass; verified live — the previously crashing configuration now serves a stable CQP session.The branch is self-contained: it carries only the CQP probe variant. If #26 (which adds the VBR probe) lands as well, the two tests coexist; git may ask for a trivial context rebase on whichever merges second.