[fast-ut] [reduced-it] [SkipRecovery] Restore Delta REORG coverage on Spark 4.1.1+ [databricks]

[wjxiz1992]

JaCoCo sql-plugin line coverage: not measurable locally — the recovered Spark 4.0/4.1 classfiles do not match the shim350 nightly aggregate

Contributes to delta-io/delta#5690.

Description

Narrow the Delta REORG regression guard from every Spark 4.0.1+ runtime to
Spark 4.0.1 through 4.0.x. This restores the existing deletion-vector
materialization scan-split test on supported Spark 4.1.1+ profiles, where
Delta 4.1.0 contains the upstream fix.

The guard is intentionally retained for Spark 4.0.1 through 4.0.x because
Delta 4.0.0 still reproduces the upstream binary incompatibility during CPU
table setup. The target remains independently skipped on Databricks, so no
Databricks execution claim is made.

Validation

• Static gates: py_compile, git diff --check, and current exemption scan passed.
• Spark 4.0.1 / Delta 4.0.0:
  • committed guard: 1 skipped, 250 deselected;
  • temporary unmask: CPU setup reproduces ParquetToSparkSchemaConverter five-argument NoSuchMethodError.
• Spark 4.0.0 / Delta 4.0.0:
  • build: BUILD SUCCESS, 14/14 reactor modules;
  • final target: 1 passed, 250 deselected;
  • plan diagnostics: FileSourceScanExec will run on GPU.
• Spark 4.1.1 / Delta 4.1.0:
  • build: BUILD SUCCESS, 14/14 reactor modules;
  • final target: 1 passed, 250 deselected;
  • full delta_lake_test.py: 248 passed, 3 skipped, 0 failed;
  • JUnit: tests=251, errors=0, failures=0, skipped=3; target passed;
  • plan diagnostics: FileSourceScanExec will run on GPU in exact and full-file runs.
• NT local review: 6 independent reviewers; one duplicated line-length finding was fixed and validator-confirmed; final result 0 must-fix, 0 should-fix, 0 suggestions.

Checklists

Documentation

• Updated for new or modified user-facing features or behaviors
• No user-facing change

Testing

• Added or modified tests to cover new code paths
• Covered by existing tests
  (Please provide the names of the existing tests in the PR description.)
• Not required

Performance

• Tests ran and results are added in the PR description
• Issue filed with a link in the PR description
• Not required

[greptile-apps]

Greptile Summary

Narrows the Delta REORG regression skip so the deletion-vector materialization scan-split test runs on supported Spark 4.1.1+ profiles while remaining skipped on Spark 4.0.1 through 4.0.x.

• Imports the existing Spark 4.1.1-or-later version predicate.
• Retains the independent Databricks skip.
• Updates the skip explanation to document the Delta 4.1.0 requirement.

Confidence Score: 5/5

The PR appears safe to merge because the narrowed skip aligns with the repository’s supported Spark and Delta profile boundaries.

The existing version helper is available, Spark 4.0.x profiles remain excluded from the incompatible REORG path, Spark 4.1.1 profiles use Delta 4.1.0, and Databricks execution remains independently skipped.

Important Files Changed

Filename	Overview
integration_tests/src/main/python/delta_lake_test.py	The revised version gate matches supported profile mappings: Spark 4.0.x uses Delta 4.0.0 and remains skipped, while Spark 4.1.1 uses Delta 4.1.0 and regains coverage.

_{Reviews (1): Last reviewed commit: "test: restore Delta REORG coverage on Sp..." | Re-trigger Greptile}

[Copilot]

Pull request overview

This PR adjusts Spark-version gating in the Delta Lake integration tests to restore coverage for the Delta REORG regression guard on supported Spark 4.1.1+ profiles, while keeping the guard for affected Spark 4.0.1–4.0.x runtimes.

Changes:

• Adds is_spark_411_or_later usage in delta_lake_test.py to refine the REORG skip condition.
• Updates the skip reason text to clarify the dependency on Delta 4.1.0 availability in supported Spark 4.1.1+ profiles.

Suppressed comments (1)

integration_tests/src/main/python/delta_lake_test.py:827

• The new skip condition (is_spark_401_or_later() and not is_spark_411_or_later()) also skips Spark 4.1.0, not just Spark 4.0.1 through 4.0.x as described in the PR. Since spark_session.py already exposes is_spark_40x(), using it would match the stated intent and avoid unintentionally skipping 4.1.0 environments.

@pytest.mark.skipif(is_spark_401_or_later() and not is_spark_411_or_later(),