Add common unshim packaging tooling [databricks]

[gerashegalov]

Related to #14834.

Description

This is the first PR in the unshim stack. The current take is to make this bottom layer infrastructure-only so it is reviewable and deliverable on its own.

This PR adds the packaging, diagnostics, build fast path, and documentation needed to evaluate common-class unshimming, but it does not move runtime/source classes and does not change normal production packaging behavior by default. The automatic promotion path for identical spark-shared classes is opt-in for local validation with UNSHIM_PROMOTE_DEFAULT_SPARK_SHARED_CLASSES=1. With that variable unset, packaging continues to use the existing explicit promotion lists.

Follow-up PRs in the stack will apply the source and packaging migrations in smaller themed batches.

Why this layer exists

The stack is moving toward an inverse unshim model: common bytecode that is identical across selected shims should be eligible for root-layout packaging, while true exceptions remain explicit.

This PR does not flip that policy on for normal builds. Instead, it adds the mechanics needed to review and validate the policy safely:

• candidate discovery and dependency analysis;
• explicit keep lists for classes that must remain shim-packaged;
• packaging diagnostics that explain why classes moved or stayed put;
• a cheaper local build path for repeated validation;
• documentation for the workflow and guardrails.

Keeping this PR infra-only avoids mixing the review of packaging mechanics with source-level bridge changes or runtime behavior changes.

What changes in this PR

• Adds an opt-in common-class promotion path for parallel-world packaging experiments.
• Adds dist/keep-in-spark-shared.txt for classes that are identical but must remain under spark-shared.
• Adds dist/keep-in-spark-shim-dirs.txt for classes that must remain in selected Spark shim directories because they need same-loader visibility to shim-only classes.
• Adds dist/scripts/analyze-parallel-world-deps.py to inspect class-file dependencies and report static paths from root/common classes to version-specific shim bytecode.
• Adds dist/scripts/build-unshim-parallel-world.py to build/package a single-shim view cheaply while iterating on candidates.
• Extends dist/scripts/binary-dedupe.sh diagnostics so the packaging result can be explained from class-file evidence.
• Optimizes build/buildall for repeated unshim iteration, including the cheaper fast path and -Ddist.jar.compress=false on that path.
• Updates docs/dev/shims.md, docs/dev/shimplify.md, docs/dev/parallel-world-unshimming.md, and dist/README.md for the new workflow.

What is intentionally not in this PR

• No runtime/source class moves.
• No selected-shim bridge refactor.
• No default production flip to automatically promote all identical common classes.
• No Databricks-specific source cleanup; DBR-sensitive changes stay isolated in follow-up PRs.

Reviewer guidance

Please review this as the infrastructure layer for the stack. The key questions are:

• Are the keep-list semantics clear and narrow enough for follow-up PRs?
• Does the analyzer provide useful evidence for deciding whether a class can move to root layout?
• Are the binary-dedupe diagnostics understandable when promotion is enabled locally?
• Is the build fast path appropriate for repeated local validation?
• Is the documentation clear that normal packaging is behavior-preserving unless the opt-in environment variable is set?

Stack map

• Add common unshim packaging tooling [databricks] #15025: packaging scaffolding, analyzer tooling, binary-dedupe diagnostics, buildall fast path, and docs.
• Add SQL plugin helper module wiring #15026: Maven/module wiring for Java-friendly SQL plugin helper modules.
• Move shared API and shuffle format helpers to Java modules #15040 and Move Hadoop file I/O helpers to Java module #15030: shared API/format helpers and Hadoop file I/O helpers.
• Add Spark 3.3.0 SQL shim module sources #15043, Add Spark 3.3.1 through 3.4 SQL shim module sources #15041, and Add Spark 3.5 and 4.0 SQL shim module sources #15042: Spark-version SQL shim module source population.
• Move column vector and host memory helpers to the columnar module #15031, Move columnar scalar utilities to Java modules #15044, Move columnar runtime config helpers to Java #15032, Move columnar table value helpers to Java modules #15045, Move profile and write stats to the columnar module #15046, and Move aggregate and shuffle stats to the columnar module #15047: columnar/vector/runtime/table/stat helper moves.
• Adapt rule and plugin metadata callers to Java helpers #15048, Adapt Kudo and file scan callers to Java helpers #15049, Adapt shuffle and spill callers to Java helpers #15050, Adapt core execution callers to columnar helpers #15051, Adapt SQL RAPIDS callers to columnar helpers #15033, Adapt remaining SQL plugin callers to Java helpers #15034, and Move common shim helpers to shared sources #15052: caller migration and common shim helper moves.
• Remove old Spark 3.3.0 shim sources #15035, Remove old Spark 3.3 DBR shim sources #15053, Remove old Spark 3.3.1 through 3.4 shim sources #15036, Remove old Spark 3.5 shim sources #15037, and Remove old Spark 4 shim sources and update tests #15038: old shim source cleanup, with the DBR cleanup isolated in Remove old Spark 3.3 DBR shim sources #15053.
• Adapt Delta and Iceberg to unshimmed helpers #15027: Delta and Iceberg follow-up adaptation.
• Update UDF compiler for shared helper layout #15028: UDF compiler and documentation follow-up adaptation.

Testing and validation notes

Local checks for the current infra-only shape:

• git diff --check
• bash -O extglob -n build/buildall
• bash -n dist/scripts/binary-dedupe.sh
• python3 -m py_compile dist/scripts/analyze-parallel-world-deps.py dist/scripts/build-unshim-parallel-world.py dist/build/package-parallel-worlds.py
• mvn verify -P individual,pre-merge -Dbuildver=330 -Ddist.jar.compress=false -DskipTests -Dmaven.scaladoc.skip -Dmaven.artifact.threads=10 --batch-mode -Dmaven.wagon.http.retryHandler.count=3 -Dmaven.wagon.httpconnectionManager.ttlSeconds=30 -Daether.connector.http.connectionMaxTtl=30

Checklists

Documentation

• Updated for new or modified user-facing features or behaviors
• No user-facing change

Testing

• Added or modified tests to cover new code paths
• Covered by existing tests
  (Covered by packaging/no-tests build validation and syntax/format checks for the new tooling.)
• Not required

Performance

• Tests ran and results are added in the PR description
• Issue filed with a link in the PR description
• Not required

[gerashegalov]

build

[gerashegalov]

build

[gerashegalov]

build

[gerashegalov]

build

[gerashegalov]

Blossom is currently blocked by known issue #15293, not by this PR's current diff.

The failing Blossom run hit RegularExpressionTranspilerSuite / string split fuzz - anchor focused with the same mismatch tracked in #15293:

string_split java pattern=\S\r+?\n*
data=\rbaab\r\ra
CPU [2]: \rbaa, \ra
GPU [2]: \rbaa, a

This PR's effective diff is now infra/docs/build tooling only (build/, dist/, docs/), so it does not touch the regex implementation or the failing test. The issue notes this is caused by the moving JNI/cuDF snapshot and is also affecting unrelated PRs.

[gerashegalov]

build

[nvauto]

NOTE: release/26.08 has been created from main. Please retarget your PR to release/26.08 if it should be included in the release.

[res-life]

IMO, the PR stack(20+) is mainly a tool?
The tool doesn't need review precisely because its output gets independently verified.
The final goal is removing paralle class loader.
Could we keep the tool in personal local, and let the tool generate new PRs.

From AI:
Why tooling like this shouldn't be in the review pipeline at all

The analyzer is a candidate generator, not a correctness proof. If it says "these 40 classes are root-safe" and it's wrong, the consequence isn't bad code landing — it's:

• binary-dedupe.sh's identity check fails the build (non-identical bytecode is caught mechanically)
• integration tests fail
• the reviewer reading the source diff notices this batch doesn't belong together

In other words, correctness comes from the downstream gate, not from the tool. Nobody reviews your grep invocations for the same reason.

There's a nice inversion here: the moment you merge it and wire it into CI, it becomes a gate — and only then does it genuinely need review, because its correctness starts bearing load. So the act of merging is what creates the review burden; not merging eliminates it entirely. The current PR sits on the wrong side of that line: it packages a non-load-bearing heuristic as something reviewers are expected to vouch for — the Tarjan implementation, the constant-pool tag dispatch, the reversed edges in the Kahn sort. In practice no reviewer can verify any of that in reasonable time. The zero test coverage I flagged earlier is the symptom of exactly this mismatch.

Migration PRs really are an order of magnitude cheaper to review

Per batch, the reviewer only needs to check:

1. Source moves under git rename detection — does this grouping make architectural sense?
2. If a keep-list or identity exception entry was added — is the stated reason credible?
3. CI green

The first two are judgments humans are actually good at and must make; the third is the machine's job. None of it requires trusting the tool.

Side benefit: the author iterates on the tool with zero process friction. Across 25 batches you inevitably learn and adjust as you go — if the tool lives in the repo, every improvement is an extra PR, and the hardest kind to review at that.

One real risk, unrelated to tooling

For this to work, every migration PR must build and test independently. But looking at the stack map, the dependency chain runs deep: #15026 wires modules → #15040/#15030 add helpers → #15043 et al. populate shim module sources → #15048 et al. migrate callers → #15035 et al. clean up old sources. If PRs 3 through 25 only make sense chained together and won't build standalone, then the "each PR is easy to review" premise is already discounted — and that has nothing to do with the tooling question. It's structural to the stack.

Worth getting explicit confirmation from the author: after each PR merges to main individually, are build and tests green? If not, the fix isn't an infrastructure PR underneath — it's redrawing the batch boundaries.

One concrete suggestion

Put the evidence for exception entries in the PR description, rather than putting the tool in the repo.

For example, when a migration PR needs an identity exception for RapidsErrorUtils, the author uses the local tool to generate a javap -c diff between the two shim versions, pastes it into the PR body, and states that the divergence is confined to Scala metadata and debug attributes with identical executable bytecode. The reviewer judges the evidence, not the tool's source.

That's the right shape for this: output as evidence in the PR, not code as an asset in the repo.

[res-life]

posted some comments from AI

[gerashegalov]

On keeping the tools in the repo: I think they should stay in this base PR because the refactor is intentionally gradual. The migration is not a one-shot local author workflow; follow-up PRs will need repeatable candidate generation, evidence collection, and packaging verification over time.

Keeping the scripts in-tree makes that process reproducible for other contributors and reviewers, keeps the command line stable across batches, and avoids making one local checkout the hidden source of truth. The tools are still advisory where appropriate: the correctness boundary remains the packaging checks, class identity checks, CI, and the source diffs in each migration PR.

[res-life]

LGTM

[gerashegalov]

build

[abellina]

AI pointed out that buildvers is sorted in build-unshim-parallel-world.py with reverse=True, but here we pick the 0th element of the item provided by user. Should we sort both lists the same way? Does it really matter?
Sorry no full context, AI and me could be wrong.

[NvTimLiu]

LGTM from CI point of view, for buildall script we'll nothing changed if we keep UNSHIM_PARALLEL_WORLD_ONLY unset

I'm not familiar with the paralele/dedupe related scripts.

Are we going to move all the common classes from spark-shared/com/nvidia to com/nvidia ?

[gerashegalov]

Are we going to move all the common classes from spark-shared/com/nvidia to com/nvidia ?

yes, eventually after a series of refactoring PRs, all classes (99%) that remain in the sql-plugin module are proven to be Spark-version independent and can reside in the standard root part of the jar under com/nvidia, org/apache.

Next PRs would work towards the new default UNSHIM_PROMOTE_DEFAULT_SPARK_SHARED_CLASSES=1

[gerashegalov]

build

[gerashegalov]

build