Skip to content

hwdec_cuda/vulkan: fix partial failure edge cases in error paths - #18017

Closed
philipl wants to merge 0 commit into
mpv-player:masterfrom
philipl:unslopped-hwdec-fixes
Closed

hwdec_cuda/vulkan: fix partial failure edge cases in error paths#18017
philipl wants to merge 0 commit into
mpv-player:masterfrom
philipl:unslopped-hwdec-fixes

Conversation

@philipl

@philipl philipl commented May 23, 2026

Copy link
Copy Markdown
Member

These changes were inspired by the slopbot, but I went through and manually verified them, identifying additional problems, and better solutions along the way.


hwdec_cuda: fix partial failure edge cases in error paths

The slopbot spam did find valid issues in the error paths in this code, and as
part of verifying them, I have additional fixes.

  • ensure that the return value -1 from cuda_init() if ext_init() fails
  • zero alloc the ext_vk and ext_gl structs so that we have a known initial
    state
  • explicitly initialise the sem_handle.fd to -1 to ensure we don't
    accidentally close stdin in the error path. But also don't do this on
    windows as an invalid Handle is 0.
  • Don't do a double cuCtxPopCurrent() in the ext_gl init failure path

Fixes #17976, #17988, #17989


hwdec_vulkan: fix partial failure edge cases in error paths

This is another file where the slopbot identified a problem, but I was
surprised to see it fix it in a poor way and also miss an adjacement
problem.

mapper_map calls mapper_unmap with the intention of using it to clean
up partial failure after some planes are wrapped and released while others
are not. But the current code would actually segfault in that situation.

There are two fixes required:

  • Ensure p->vkf is set by mapper_map early so that its available to
    mapper_unmap in the error path
  • Don't call vkfc->unlock_frame in the mapper_map error path, as
    this will be called by mapper_unmap

Fixes #17977


@philipl
philipl force-pushed the unslopped-hwdec-fixes branch from 826d640 to 7858bf8 Compare May 23, 2026 18:01

@kasper93 kasper93 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks!

@philipl philipl closed this May 27, 2026
@philipl
philipl force-pushed the unslopped-hwdec-fixes branch from 7858bf8 to d05814c Compare May 27, 2026 01:23
@philipl
philipl deleted the unslopped-hwdec-fixes branch May 27, 2026 01:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] hwdec_vulkan: NULL pointer dereference in mapper_unmap when p->vkf is NULL [Bug] hwdec_cuda: mapper_init returns 0 on ext_init failure

2 participants