Add parallel native library extraction

[gerashegalov]

Description

Add an internal chunk-manifest resource format for large native libraries and extract those chunks concurrently. The loader:

• preserves conventional single-resource loading and ai.rapids.cudf.lib-native-dir
• reads up to 12 JAR entries concurrently
• writes directly to the pre-sized output with positional FileChannel writes
• validates manifest structure, exact chunk sizes, and per-chunk CRC32 values
• removes partial output after extraction failures

Related issue: NVIDIA/cudf-spark#15145 identifies synchronous, single-threaded native JAR extraction as a major executor startup cost.

This allows distribution JARs containing 900+ MiB CUDA native libraries to trade a measured 0.0087% increase in archive size for substantially faster startup and JAR creation. NVIDIA/cudf-spark#15356 is the companion producer PR.

Performance

Native extraction

Measured the exact pre-change and PR paths on the same host with JDK 17 and the same 1,504,051,352-byte CUDA12 libcudf.so. Each case ran six extractions in one JVM; the first invocation was treated as warm-up and the remaining five were used for the median.

Path	First invocation	Post-warm-up range	Median
Pre-change conventional DEFLATED resource, 16 KiB sequential copy	6.411 s	6.370-6.426 s	6.371 s
45 DEFLATED chunks, 12-worker positional extraction	1.164 s	1.078-1.124 s	1.113 s

The post-warm-up median improves by 5.72x, reducing extraction latency by 82.5%. The first measured invocation improves by 5.51x. Both paths produced exactly 1,504,051,352 bytes.

Archive size and creation time

Using matched resource trees and forcing the existing jar:jar@create-parallel-worlds-jar execution:

Input representation	Median	Output JAR size
One conventional DEFLATED libcudf.so entry	55.07 s	952,504,790 bytes
45 DEFLATED chunks plus manifest	5.47 s	952,587,551 bytes

Chunking increased the archive by 82,761 bytes (0.0087%) while making focused JAR creation 10.07x faster, a 90.1% wall-time reduction. The full benchmark methodology and measured ranges are in NVIDIA/cudf-spark#15356.

Validation

• mvn compiler:compile compiler:testCompile
• mvn surefire:test -Dtest=NativeDepsLoaderExtractionTest (10 tests)
• mvn surefire:test@native-deps-loader-test (5 tests)
• Cross-repository production JAR benchmark described above

The full native Maven lifecycle was not run locally because the installed CMake is 3.28 and this branch requires CMake 4.0; the Java sources and focused tests were compiled and run directly.

Checklist

• I am familiar with the Contributing Guidelines.
• New or existing tests cover these changes.
• The documentation is up to date with these changes.

[coderabbitai]

📝 Walkthrough

Walkthrough

NativeDepsLoader now supports conventional and chunked native resource extraction. Chunked resources are parsed from manifests, extracted concurrently into pre-sized files, verified with size and CRC32 checks, and cleaned up on failure. Tests cover successful extraction, concurrency, malformed resources, corruption, and empty library directories.

Changes

Native dependency extraction

Layer / File(s)	Summary
Extraction selection and dependency validation java/src/main/java/ai/rapids/cudf/NativeDepsLoader.java, java/src/test/java/ai/rapids/cudf/NativeDepsLoaderTest.java	Configured native directories now validate the requested dependency, and createFile selects conventional or chunked resources with partial-file cleanup.
Concurrent chunk extraction and integrity checks java/src/main/java/ai/rapids/cudf/NativeDepsLoader.java	Chunk manifests are validated, chunks are written concurrently at random-access offsets, and output size and CRC32 values are checked.
Extraction behavior and failure coverage java/src/test/java/ai/rapids/cudf/NativeDepsLoaderExtractionTest.java	Tests cover conventional and chunked extraction, concurrent readers, invalid manifests, missing or truncated chunks, CRC mismatches, and cleanup of failed outputs.

Estimated code review effort: 4 (Complex) | ~45 minutes

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	The title accurately summarizes the main change: concurrent native library extraction.
Description check	✅ Passed	The description matches the changeset and explains chunked concurrent extraction, validation, and failure handling.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands.}

[zpuller]

"trade a small amount of archive compression for substantially faster startup" can you quantify this? Are you saying the jar is slightly larger?

[gerashegalov]

"trade a small amount of archive compression for substantially faster startup" can you quantify this? Are you saying the jar is slightly larger?

In the matched benchmark NVIDIA/cudf-spark#15356 , the conventional archive was 952,504,790 bytes and the chunked archive was 952,587,551 bytes: an increase of 82,761 bytes, or 0.0087%.

The corresponding focused JAR creation median improved from 55.07 seconds to 5.47 seconds.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[ttnghia]

Is there any potential race issue with this parallel execution model? If so, we may need to pay more attention to the relevant functions that run in parallel.

[gerashegalov]

Is there any potential race issue with this parallel execution model? If so, we may need to pay more attention to the relevant functions that run in parallel.

I am trying to pay attention, the input chunks and the output ranges within a file do not overlap. "File channels are safe for use by multiple concurrent threads."

[gerashegalov]

/ok to test 2cb0c29

[mythrocks]

@gerashegalov: It appears your change has survived this round of Russian roulette. :] CI has passed.

[gerashegalov]

/merge