udev: prevent interruption by thread cancellation by wuguanghao3 · Pull Request #24 · openSUSE/multipath-tools

wuguanghao3 · 2026-07-20T09:18:13Z

Using mutexes only guarantees no concurrency during normal runtime. However, if a thread is cancelled mid-execution, a libudev API call might be interrupted halfway (partially executed), leaving memory values in an inconsistent or abnormal state.

Furthermore, this change aligns with the requirement that libudev interfaces are not thread-safe and must not be called from multiple threads.

Using mutexes only guarantees no concurrency during normal runtime. However, if a thread is cancelled mid-execution, a libudev API call might be interrupted halfway (partially executed), leaving memory values in an inconsistent or abnormal state. Furthermore, this change aligns with the requirement that libudev interfaces are not thread-safe and must not be called from multiple threads. Signed-off-by: Wu Guanghao <wuguanghao3@huawei.com>

mwilck · 2026-07-20T09:41:42Z

Thanks! I have applied your patch to my tip branch, and applied another patch on top to wrap all libudev calls with macros, as proposed in opensvc#153.

mwilck · 2026-07-20T09:43:46Z

@wuguanghao3, @bmarzins, may I ask you to review 58519b1 ?

wuguanghao3 · 2026-07-20T11:25:23Z

@wuguanghao3, @bmarzins, may I ask you to review 58519b1 ?

LGTM

But I have a question: can't we just use (call) directly instead of udev_xxx(arg1, arg2, ...)? Do we need to use different macros based on the parameters? What is the advantage of doing it this way?

#define MT_WRAP(call) { \
   ...\
   __ret = (call);  
   ...\
}

MT_WRAP((udev_xxx)(a))
MT_WRAP((udev_yyy)(a,b))
...

mwilck · 2026-07-21T08:03:37Z

But I have a question: can't we just use (call) directly instead of udev_xxx(arg1, arg2, ...)?

In principle we could. We wouldn't need the mt_udev... wrappers at all, we'd just wrap every call in-place. I recall that we had some weird problems with nested pthread_cleanup_push() calls with some old compilers in the past.

Let me try.

mwilck · 2026-07-21T08:45:40Z

I remember the rationale now. @wuguanghao3, your suggestion works only with preprocessor macros. For wrapper functions it's either impossible or really clumsy, because we need to insert the argument types in the wrapper function's argument list, while omitting them in the actual libudev call.

(If you can conceive of an elegant way to do this, let me know).

What we can do is just use preprocessor macros:

#define MT_UDEV_WRAP(call)						\
	({								\
		int __oldstate;						\
		pthread_setcancelstate(PTHREAD_CANCEL_DISABLE, &__oldstate); \
		pthread_mutex_lock(&libudev_mutex);			\
		__typeof__(call) __ret = (call);			\
		pthread_mutex_unlock(&libudev_mutex);			\
		pthread_setcancelstate(__oldstate, NULL);		\
		pthread_testcancel();					\
		__ret;							\
	})

#define udev_ref(udev) MT_UDEV_WRAP(udev_ref(udev))
#define udev_list_entry_get_by_name(list_entry, name) \
        MT_UDEV_WRAP(udev_list_entry_get_by_name(list_entry, name))
...

This works, and we don't even need any mt_udev_ wrapper symbols. We just need to make libudev_mutex globally visible.

However, we have the wrapper functions for a reason. By introducing the wrapper functions, we achieve clean separation from libudev, and we can make sure that we don't ever call libudev directly. We just don't include libudev.h anywhere except in mt-libudev.c¹. When we use the preprocessor-only approach, we have much less separation; we need to include libudev.h basically everywhere, and make libudev_mutex globally visible.
Moreover, I don't like inlining the cancellation prevention code in every libudev call; IMO it's an example of "bad" inlining. But that's mainly a matter of taste.

This makes me realize that we've actually got this wrong in some recent commits! ↩

mwilck · 2026-07-21T09:06:54Z

I realize what I did previously here was wrong. By applying my commit on top of yours, I called pthread_setcancelstate() twice (actually 4 times!) now for every udev call.

I've pushed a new version to my tip branch (b2d35f7). @wuguanghao3, please review test it in your environment.

And again, if you can conceive of a simpler way to write these wrappers, let me know.

mwilck · 2026-07-21T10:13:23Z

I had to rebase and re-push again, current commit ID is 3019b25

bmarzins

Looks good to me.

wuguanghao3 · 2026-07-22T02:13:51Z

I had to rebase and re-push again, current commit ID is 3019b25

OK, LGTM

mwilck · 2026-07-22T08:23:37Z

@wuguanghao3

OK, LGTM

Have you run your test with this code?

wuguanghao3 · 2026-07-23T02:37:35Z

@wuguanghao3

OK, LGTM

Have you run your test with this code?

Yes, currently testing. The issue has not recurred yet.

wuguanghao3 · 2026-07-23T07:17:57Z

@wuguanghao3

OK, LGTM

Have you run your test with this code?

Yes, currently testing. The issue has not recurred yet.

The UAF issue could not be reproduced. However, a new reference issue occurred, which appears to be exclusive to version 0.14.3 and was not present in 0.9.5. I am currently running tests on 0.14.3 with the current patch applied.

(gdb) bt
#0  0x00007f27a3b97bac in ?? () from /usr/lib64/libc.so.6
#1  0x00007f27a3b4bb86 in raise () from /usr/lib64/libc.so.6
#2  0x00007f27a3b363fc in abort () from /usr/lib64/libc.so.6
#3  0x00007f27a3dfd8ca in log_assert_failed (text=<optimized out>, file=<optimized out>, line=<optimized out>, func=<optimized out>) at ../src/basic/log.c:976
#4  0x00007f27a3e00507 in udev_device_unref (p=<optimized out>) at ../src/libudev/libudev-device.c:508
#5  udev_device_unref (p=p@entry=0x556220ff4d70) at ../src/libudev/libudev-device.c:508
#6  0x00007f27a3e362af in mt_udev_device_unref (__arg1=0x556220ff4d70) at mt-libudev.c:80
#7  0x00007f27a3ebff5d in free_path (pp=0x556220ffa060) at structs.c:179
#8  0x00007f27a3ebfffd in free_pathvec (vec=0x556220db2d90, free_paths=<optimized out>) at structs.c:201
#9  0x000055620cde21b8 in cleanup_paths (vecs=0x556220d90620) at main.c:3792
#10 cleanup_vecs () at main.c:3807
#11 cleanup_child () at main.c:3916
#12 0x00007f27a3b4e135 in ?? () from /usr/lib64/libc.so.6
#13 0x00007f27a3b4e230 in exit () from /usr/lib64/libc.so.6
#14 0x00007f27a3b37617 in ?? () from /usr/lib64/libc.so.6
#15 0x00007f27a3b376c9 in __libc_start_main () from /usr/lib64/libc.so.6
#16 0x000055620cde15d5 in _start ()

mwilck · 2026-07-23T09:30:13Z

Sigh. This indicates a refcount underflow on the udev_device. Which is basically the same thing as a UAF.

Could you perhaps apply the current commit to 0.9.5 and see if it's really a regression between 0.95 and 0.14.3? Given that the issue has apparently occured after 4h of testing, you proabably need to run the test for a couple of hours to verify that the problem does not occur.

wuguanghao3 · 2026-07-24T02:09:00Z

Sigh. This indicates a refcount underflow on the udev_device. Which is basically the same thing as a UAF.

Could you perhaps apply the current commit to 0.9.5 and see if it's really a regression between 0.95 and 0.14.3? Given that the issue has apparently occured after 4h of testing, you proabably need to run the test for a couple of hours to verify that the problem does not occur.

The UAF and the reference issue are distinct. The UAF involves memory inconsistency resulting from thread cancellation, whereas the reference issue may stem from a redundant execution of an upper-level call.
During a 24-hour test of the initial patch on version 0.9.5, the issue could not be reproduced. Given that the underlying logic remains identical between the final and initial versions, there is a high probability that this issue was introduced in a release between 0.9.5 and 0.14.3. I will conduct further testing on version 0.9.5 utilizing the latest proposed solution.

wuguanghao3 · 2026-07-24T08:38:56Z

Sigh. This indicates a refcount underflow on the udev_device. Which is basically the same thing as a UAF.
Could you perhaps apply the current commit to 0.9.5 and see if it's really a regression between 0.95 and 0.14.3? Given that the issue has apparently occured after 4h of testing, you proabably need to run the test for a couple of hours to verify that the problem does not occur.

The UAF and the reference issue are distinct. The UAF involves memory inconsistency resulting from thread cancellation, whereas the reference issue may stem from a redundant execution of an upper-level call. During a 24-hour test of the initial patch on version 0.9.5, the issue could not be reproduced. Given that the underlying logic remains identical between the final and initial versions, there is a high probability that this issue was introduced in a release between 0.9.5 and 0.14.3. I will conduct further testing on version 0.9.5 utilizing the latest proposed solution.

It has been confirmed that this issue was introduced in a release between 0.9.5 and 0.14.3. I conducted a 7-hour test running version 0.9.5 with the latest commit, and the issue could not be reproduced.

mwilck · 2026-07-24T10:08:37Z

The UAF involves memory inconsistency resulting from thread cancellation, whereas the reference issue may stem from a redundant execution of an upper-level call.

A refcount underflow means that an object is being unref'd after having reached refcount zero. IOW, this object was still in use after it should have been freed already. Maybe the free() hadn't happened yet, or for some reason libasan didn't catch the problem. IMO it is pure "luck" that the memory where the refcount is stored is still accessible and contains a negative value.

How often have you observed this refcount underflow so far?
Do you have log files (ideally with verbosity 3) where we can see what happened before the crash?

It has been confirmed that this issue was introduced in a release between 0.9.5 and 0.14.3

There has been a major cleanup of the code that frees maps and paths in 0.14.0 (commit edf2bac ff.) So if you want to to investigate further, you may want to start with comparing 0.13.0 and 0.14.0, both with the latest wrapper patch applied. When you apply the patch, make sure that no direct libudev calls remain.

Given that you observe the problem in the exit() code flow, while multipathd frees its internal data structures, the easiest fix would be to just skip the exit handlers, the only purpose of which is to show zero memory leaks when multipathd is run under a leak checking program. This would have no negative consequences for running multipathd in production, because the kernel frees the entire memory used by the process anyway. But it's not a solution we can generally apply, because it would strongly affect our ability to assess multipathd for memory leaks. In theory, we could just skip the call to cleanup_vecs() in the atexit handler in "production" builds. But obviously I'd prefer to find the actual problem.

Can you share your test program, so that others can try to reproduce the issue and find a solution? I have also done some extensive testing, but apparently the way you restart multipathd during the test can trigger error conditions that I was not seeing.

wuguanghao3 · 2026-07-25T01:15:45Z

The UAF involves memory inconsistency resulting from thread cancellation, whereas the reference issue may stem from a redundant execution of an upper-level call.

A refcount underflow means that an object is being unref'd after having reached refcount zero. IOW, this object was still in use after it should have been freed already. Maybe the free() hadn't happened yet, or for some reason libasan didn't catch the problem. IMO it is pure "luck" that the memory where the refcount is stored is still accessible and contains a negative value.

How often have you observed this refcount underflow so far? Do you have log files (ideally with verbosity 3) where we can see what happened before the crash?

It has been confirmed that this issue was introduced in a release between 0.9.5 and 0.14.3

There has been a major cleanup of the code that frees maps and paths in 0.14.0 (commit edf2bac ff.) So if you want to to investigate further, you may want to start with comparing 0.13.0 and 0.14.0, both with the latest wrapper patch applied. When you apply the patch, make sure that no direct libudev calls remain.

Given that you observe the problem in the exit() code flow, while multipathd frees its internal data structures, the easiest fix would be to just skip the exit handlers, the only purpose of which is to show zero memory leaks when multipathd is run under a leak checking program. This would have no negative consequences for running multipathd in production, because the kernel frees the entire memory used by the process anyway. But it's not a solution we can generally apply, because it would strongly affect our ability to assess multipathd for memory leaks. In theory, we could just skip the call to cleanup_vecs() in the atexit handler in "production" builds. But obviously I'd prefer to find the actual problem.

Can you share your test program, so that others can try to reproduce the issue and find a solution? I have also done some extensive testing, but apparently the way you restart multipathd during the test can trigger error conditions that I was not seeing.

Below is a simplified test case that includes the main operations. One additional point to note: we identify local disks as multipath by default.

CONFIG_CONTENT='devices {
    device {
        vendor "(LIO-ORG)"
        product ".*"
        no_path_retry "fail"
    }
}'

function prepare_local_target() {
    local count=100
    local testdir=$1
    mkdir -p "$testdir"
    i=1
    while [ $i -le $count ]; do
        dd if=/dev/zero of="$testdir"/disk$i count=10 bs=1M
        ((i = i + 1))
    done

    i=1
    echo -e "clearconfig confirm=true\n" | targetcli
    echo -e "cd /backstores/fileio\n" | targetcli
    while [ $i -le $count ]; do
        echo -e "create file_or_dev=$testdir/disk$i name=filedisk$i size=100M\n" | targetcli
        ((i = i + 1))
    done
    echo -e "cd /iscsi\ncreate iqn.2003-01.org.linux-iscsi.localhost.test:sn.ff69cf92aac7\n" | targetcli
    echo -e "cd iqn.2003-01.org.linux-iscsi.localhost.test:sn.ff69cf92aac7/tpg1/luns\n" | targetcli
    i=1
    while [ $i -le $count ]; do
        echo -e "create /backstores/fileio/filedisk$i lun=$i\n" | targetcli
        ((i = i + 1))
    done
    echo -e "cd /iscsi/iqn.2003-01.org.linux-iscsi.localhost.test:sn.ff69cf92aac7/tpg1/acls\n" | targetcli
    echo -e "create iqn.2003-01.org.linux-iscsi.localhost.test:sn.ff69cf92aac7:client\n" | targetcli
    systemctl restart target

    echo "InitiatorName=iqn.2003-01.org.linux-iscsi.localhost.test:sn.ff69cf92aac7:client" > /etc/iscsi/initiatorname.iscsi
    systemctl restart iscsid
}

function udev_trigger() {
    while true; do
        for i in $(seq 1 20); do
            udevadm trigger
        done
        sleep 1
    done
}

function pre_test() {
    yum install -y targetcli open-iscsi multipath-tools target-restore scsi-target-utils || exit "${ETS_FAILED}"
    cp /etc/iscsi/initiatorname.iscsi .
    prepare_local_target /home/fileio-$$

    iscsiadm -m node --logout ALL
    iscsiadm -m node --op delete ALL
    iscsiadm -m discovery -p 127.0.0.1 -t st || exit "${ETS_FAILED}"
    iscsiadm -m node -p 127.0.0.1 -l || exit "${ETS_FAILED}"
    echo remove_local_disk=1 > /etc/multipath_private.conf
    version_compare "$(rpm -q --qf '%{version}' multipath-tools)" ">=" "0.9.5" && echo "$CONFIG_CONTENT" > /etc/multipath.conf
}

function do_test() {
    ulimit -c unlimited
    version_compare "$(rpm -q --qf '%{version}' multipath-tools)" ">=" "0.9.5" && modprobe dm-multipath
    systemctl restart multipathd || exit "${ETS_FAILED}"
    udev_trigger &> /dev/null &
    pid_udev=$!

    time=$(date "+%Y-%m-%d %H:%M:%S")
    for loop in $(seq 1 20); do
        for i in $(seq 1 5); do
            systemctl restart multipathd
        done
        kill -9 "$(pidof /sbin/multipathd)"

        coredumpctl list -S "$time" | grep multipathd
        if [ $? -eq 0 ]; then
            coredumpctl info
            kill -9 "$pid_udev"
            add_failure "multipathd segfault"
            return "${ETS_FAILED}"
        fi
    done
    kill -9 "$pid_udev"
}

function post_test() {
    iscsiadm -m node -p 127.0.0.1 -u
    iscsiadm -m node -o delete -p 127.0.0.1
    multipath -F
    systemctl stop multipathd
    systemctl stop iscsid
    systemctl stop target
    cp initiatorname.iscsi /etc/iscsi/initiatorname.iscsi
    rm -rf /home/fileio-$$
}

wuguanghao3 mentioned this pull request Jul 20, 2026

udev: prevent interruption by thread cancellation opensvc/multipath-tools#153

Open

bmarzins approved these changes Jul 21, 2026

View reviewed changes

Uh oh!

Conversation

wuguanghao3 commented Jul 20, 2026

Uh oh!

mwilck commented Jul 20, 2026

Uh oh!

mwilck commented Jul 20, 2026

Uh oh!

wuguanghao3 commented Jul 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mwilck commented Jul 21, 2026

Uh oh!

mwilck commented Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Footnotes

Uh oh!

mwilck commented Jul 21, 2026

Uh oh!

mwilck commented Jul 21, 2026

Uh oh!

bmarzins left a comment

Choose a reason for hiding this comment

Uh oh!

wuguanghao3 commented Jul 22, 2026

Uh oh!

mwilck commented Jul 22, 2026

Uh oh!

wuguanghao3 commented Jul 23, 2026

Uh oh!

wuguanghao3 commented Jul 23, 2026

Uh oh!

mwilck commented Jul 23, 2026

Uh oh!

wuguanghao3 commented Jul 24, 2026

Uh oh!

wuguanghao3 commented Jul 24, 2026

Uh oh!

mwilck commented Jul 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

wuguanghao3 commented Jul 25, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

wuguanghao3 commented Jul 20, 2026 •

edited

Loading

mwilck commented Jul 21, 2026 •

edited

Loading

mwilck commented Jul 24, 2026 •

edited

Loading