Describe the bug
CUDA 12 wheel test jobs fail at import time with an undefined symbol error from pylibcudf, before any cuOpt code runs:
ImportError: /pyenv/versions/3.11.16/lib/python3.11/site-packages/pylibcudf/io/parquet_metadata.abi3.so:
undefined symbol: _ZN4cudf2io32read_parquet_column_chunk_boundsESt4spanIKNS0_7parquet12FileMetaDataE...
demangled: cudf::io::read_parquet_column_chunk_bounds(...)
The import chain that triggers it is from pylibcudf.io.experimental.hybrid_scan import ....
Root cause
The installed libcudf and pylibcudf nightlies are skewed:
libcudf-cu12-26.10.0a291.post260821111824 <- built 11:18
pylibcudf-cu12-26.10.0a292.post260821162058 <- built 16:20, ~5h later
pylibcudf a292 was built against a libcudf that exports read_parquet_column_chunk_bounds; the resolver pinned libcudf a291, which does not. Both satisfy ==26.10.*, so pip is free to pick mismatched nightlies.
Observed in
Run 32495102132:
| job |
result |
wheel-tests-cuopt / 12.2.2, 3.11, arm64, ubuntu22.04, a100, latest-deps |
fail |
wheel-tests-cuopt-server / 12.9.2, 3.14, amd64, ubuntu24.04, h100, latest-deps |
fail |
wheel-tests-cuopt / 12.9.2, 3.11, amd64, l4, oldest-deps |
pass |
wheel-tests-cuopt / 12.9.2, 3.14, amd64, h100, latest-deps |
pass |
Only latest-deps jobs are affected, which is consistent with nightly drift: oldest-deps resolves to an older, self-consistent pair.
Failure mode is a collection error, so it takes out the whole module -- one of the runs reported 0 passed, 1 failed, 20 errors.
Not cuOpt code
The failure happens while importing pylibcudf, before cuOpt is exercised. It is unrelated to whatever a given PR changes; it surfaced on an unrelated routing fix.
Suggested next steps
Describe the bug
CUDA 12 wheel test jobs fail at import time with an
undefined symbolerror frompylibcudf, before any cuOpt code runs:demangled:
cudf::io::read_parquet_column_chunk_bounds(...)The import chain that triggers it is
from pylibcudf.io.experimental.hybrid_scan import ....Root cause
The installed
libcudfandpylibcudfnightlies are skewed:pylibcudfa292 was built against alibcudfthat exportsread_parquet_column_chunk_bounds; the resolver pinnedlibcudfa291, which does not. Both satisfy==26.10.*, so pip is free to pick mismatched nightlies.Observed in
Run 32495102132:
wheel-tests-cuopt / 12.2.2, 3.11, arm64, ubuntu22.04, a100, latest-depswheel-tests-cuopt-server / 12.9.2, 3.14, amd64, ubuntu24.04, h100, latest-depswheel-tests-cuopt / 12.9.2, 3.11, amd64, l4, oldest-depswheel-tests-cuopt / 12.9.2, 3.14, amd64, h100, latest-depsOnly
latest-depsjobs are affected, which is consistent with nightly drift:oldest-depsresolves to an older, self-consistent pair.Failure mode is a collection error, so it takes out the whole module -- one of the runs reported
0 passed, 1 failed, 20 errors.Not cuOpt code
The failure happens while importing
pylibcudf, before cuOpt is exercised. It is unrelated to whatever a given PR changes; it surfaced on an unrelated routing fix.Suggested next steps
libcudf/pylibcudfto the same nightly buildlatest-depsjobs should pin matched RAPIDS nightly builds rather than==26.10.*independently