util/sysfs: libcxl fails to bind on kernels >= 6.2 due to cxl_nvdimm_bridge

[xutao323]

The kernel commit torvalds/linux@cb9cfff removed bind attributes for cxl_pmem_region and cxl_nvdimm* objects.

Current libcxl logic attempts to find a bind attribute for every entry in /sys/bus/cxl/drivers/. Since some cxl objects do not have a bind attribute anymore, libcxl fails probing with ENOENT (No such file or directory). This masks actual device errors and prevents proper enumeration on kernels v6.2 and newer.

A simple check for bind shall be able to fix and pass the right errno from other cxl modules.

diff --git a/util/sysfs.c b/util/sysfs.c
index e027e38..41ee4d6 100644
--- a/util/sysfs.c
+++ b/util/sysfs.c
@@ -216,6 +216,11 @@ int __util_bind(const char *devname, struct kmod_module *module,
                        continue;
                }
 
+               if (access(drv_path, F_OK) != 0) {
+                       free(drv_path);
+                       continue;
+               }
+
                rc = __sysfs_write_attr_quiet(ctx, drv_path, devname);
                free(drv_path);
                if (rc == 0)

[xutao323]

The background is: I was setting up test environment on linux-6.6 with CXL-2.0 type.3 hardware. The in-tree CXL drivers entered a weird state after I messed up CXL related BIOS settings.

Moduels in /sys/bus/cxl/drivers:
cxl_mem
cxl_nvdimm
cxl_nvdimm_bridge
cxl_pmem_region
cxl_port
cxl_region

libcxl were throwing ENOENT bind errors for cxl_pmem_region and cxl_nvdimm* modules as they were removed in linux-6.2 from commit cb9cfff "cxl/acpi: Simplify cxl_nvdimm_bridge probing". I skipped those non-existing bind and saw bind from other modules returned -6/ENXIO and -19/ENODEV. From a simple bpftrace script:

# ./libcxl.bt 
WARNING: Addrspace is not set
Attaching 7 probes...
Tracing cxl-cli probes.. Hit Ctrl-C to end.
TIME(us)   TID      COMM     FUNC                                               RET  
1451976    12101    cxl      uprobe:/lib/libcxl.so.1:write_attr                
----path: /sys/bus/cxl/flush, buf: 
1452009    12101    cxl      uretprobe:/lib/libcxl.so.1:write_attr              0    
1452121    12101    cxl      uprobe:/lib/libcxl.so.1:write_attr                
----path: /sys/bus/cxl/flush, buf: 
1452128    12101    cxl      uretprobe:/lib/libcxl.so.1:write_attr              0    
1452244    12101    cxl      uprobe:/lib/libcxl.so.1:write_attr                
----path: /sys/bus/cxl/flush, buf: 
1452252    12101    cxl      uretprobe:/lib/libcxl.so.1:write_attr              0    
1452284    12101    cxl      uprobe:/lib/libcxl.so.1:__util_bind               
----devname: mem0, bus: cxl
1452294    12101    cxl      uprobe:/lib/libcxl.so.1:write_attr                
----path: /sys/bus/cxl/drivers/cxl_port/bind, buf: 
1452306    12101    cxl      uretprobe:/lib/libcxl.so.1:write_attr              -19  
1452308    12101    cxl      uprobe:/lib/libcxl.so.1:write_attr                
----path: /sys/bus/cxl/drivers/cxl_mem/bind, buf: 
1532373    12101    cxl      uretprobe:/lib/libcxl.so.1:write_attr              -6   
1532384    12101    cxl      uprobe:/lib/libcxl.so.1:write_attr                
----path: /sys/bus/cxl/drivers/cxl_region/bind, buf: 
1532406    12101    cxl      uretprobe:/lib/libcxl.so.1:write_attr              -19  
1532417    12101    cxl      uretprobe:/lib/libcxl.so.1:__util_bind             -6

After fixed some BIOS settings, I was able to enable cxl device via cxl enable-memdev.

[AlisonSchofield]

@xutao323 Thanks for the report!

I instrumented __util_bind() and observed the v84 libcxl attempting writes to missing CXL bind paths on newer kernels. I saw it with cxl_pmem_region, cxl_dax_region, and cxl_nvdimm_bridge. So the underlying bind assumption is definitely broken.

One caveat is that __util_bind() never propogates that ENOENT rc value. How are you seeing libcxl probe fail with ENOENT ?

Your suggested patch looks good. This project doesn't use pull requests [1] so you can submit a patch to the mailing list or ask me to post to the list on your behalf.

[1] Patch Guidelines: https://github.com/pmem/ndctl?tab=contributing-ov-file#readme

[xutao323]

Thanks for confirming.

I would appreciate it if you could post the patch to the mailing list on my behalf. You can use my email in Reported-by and Signed-off-by: Tao Xu <xutao323@gmail.com>

And for the ENOENT, it's from subsequent open() a missing bind file:

# cxl enable-memdev -v mem0
libcxl: __sysfs_device_parse: base: '/sys/bus/cxl/devices' dev: 'mem'
libcxl: add_cxl_memdev: mem1: base: '/sys/bus/cxl/devices/mem1'
libcxl: __sysfs_device_parse: base: '/sys/bus/cxl/devices/mem1' dev: 'pmem'
libcxl: __sysfs_device_parse: mem1: processed
libcxl: add_cxl_memdev: mem0: base: '/sys/bus/cxl/devices/mem0'
libcxl: __sysfs_device_parse: base: '/sys/bus/cxl/devices/mem0' dev: 'pmem'
libcxl: __sysfs_device_parse: mem0: processed
libcxl: write_attr: failed to open /sys/bus/cxl/drivers/cxl_nvdimm/bind: No such file or directory
libcxl: write_attr: failed to open /sys/bus/cxl/drivers/cxl_pmem_region/bind: No such file or directory
libcxl: write_attr: failed to open /sys/bus/cxl/drivers/cxl_nvdimm_bridge/bind: No such file or directory
libcxl: __util_bind: mem0: bind failed
libcxl: cxl_memdev_enable: mem0: failed to enable
cxl memdev: cmd_enable_memdev: enabled 0 mem

The trigger in my case seems to be a combination of a newer kernel (6.2+) and specific BIOS CXL configurations. I also tested newer 6.19 kernel and got consistant results in my setup.