Fix OSS Delta RTAS on Spark 4.x+ [databricks]

[nartal1]

Fixes #15394

Description

On OSS Spark 4.0 with Delta Lake 4.0, CREATE OR REPLACE TABLE ... USING DELTA AS SELECT ... succeeds on CPU but fails on GPU with:

[UNSUPPORTED_FEATURE.TABLE_OPERATION] Table ... does not support truncate in batch mode

Spark 4 plans RTAS as an unconditional OverwriteByExpression. Its table capability check therefore requires the staged table to support TRUNCATE.

The RAPIDS staged Delta table advertised only V1_BATCH_WRITE, and its write builder did not implement SupportsTruncate. In addition, the Delta provider's OverwriteByExpressionExecV1 handling recognized regular DeltaTableV2 tables but not the RAPIDS staged Delta table used by atomic RTAS.

In this PR:

• Added TRUNCATE to the capabilities advertised by GpuStagedDeltaTableV2.
• Implemented SupportsTruncate in the staged Delta V1 write builder.
• Tags and converts OverwriteByExpressionExecV1 for RAPIDS staged Delta tables to GpuOverwriteByExpressionExecV1.
• Applied the staged-table handling to the Delta 3.3, 4.0, and 4.1 provider implementations.
• Preserved the existing behavior for regular DeltaTableV2 tables and continues to reject unrelated table implementations.
• Limits the existing Delta issue [FEA][Audit] - [SPARK-37822][SQL] StringSplit should return an array on non-null elements #4671 RTAS xfails to the affected Spark 3.5 releases instead of also applying them to Spark 4.
• Added an integration regression test.

Testing

OSS Spark 4.0 / Delta Lake 4.0

Focused integration test:

TEST=test_delta_rtas_truncate_capability

Result: 1 passed, 39635 deselected.

Broader focused RTAS selection:

TEST="test_delta_rtas_truncate_capability or test_delta_rtas_sql"

Result: 3 passed, 1 skipped, 39632 deselected.

Checklists

Documentation

• Updated for new or modified user-facing features or behaviors
• No user-facing change

Testing

• Added or modified tests to cover new code paths
• Covered by existing tests
  (Please provide the names of the existing tests in the PR description.)
• Not required

Performance

• Tests ran and results are added in the PR description
• Issue filed with a link in the PR description
• Not required

[greptile-apps]

Greptile Summary

The PR enables GPU execution of Spark 4 atomic Delta RTAS operations.

• Advertises staged-table truncate support and implements the corresponding write-builder interface.
• Recognizes and converts staged Delta overwrite plans across the Delta 3.3, 4.0, and 4.1 providers.
• Narrows obsolete Spark 3.5 xfails and adds a Spark 4 regression test covering execution-plan placement and resulting table contents.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains.

Important Files Changed

Filename	Overview
delta-lake/common/src/main/delta-33x-41x/scala/com/nvidia/spark/rapids/delta/GpuDeltaCatalogBase.scala	Adds the staged-table capability and write-builder contract Spark 4 requires for unconditional RTAS overwrite planning.
delta-lake/delta-33x/src/main/scala/com/nvidia/spark/rapids/delta/delta33x/Delta33xProvider.scala	Extends overwrite tagging and conversion to the staged Delta table while preserving regular DeltaTableV2 handling.
delta-lake/delta-40x/src/main/scala/com/nvidia/spark/rapids/delta/delta40x/Delta40xProvider.scala	Adds staged-table overwrite tagging and GPU conversion for the Delta 4.0 provider.
delta-lake/delta-41x/src/main/scala/com/nvidia/spark/rapids/delta/delta41x/Delta41xProvider.scala	Adds staged-table overwrite tagging and GPU conversion for the Delta 4.1 provider.
integration_tests/src/main/python/delta_lake_write_test.py	Updates Spark-version xfails and verifies Spark 4 RTAS GPU plan placement and CPU-equivalent replacement contents.

Sequence Diagram

sequenceDiagram
  participant Spark as Spark 4 Planner
  participant Catalog as GPU Delta Catalog
  participant Provider as Delta Provider
  participant Exec as GPU Overwrite Exec
  participant Commit as Staged Delta Commit
  Spark->>Catalog: Stage CREATE OR REPLACE TABLE
  Catalog-->>Spark: Staged table with TRUNCATE capability
  Spark->>Catalog: Build truncate write
  Spark->>Provider: Tag OverwriteByExpressionExecV1
  Provider-->>Spark: GPU-compatible staged table
  Spark->>Exec: Convert and execute overwrite
  Exec->>Catalog: Capture AS SELECT data
  Spark->>Commit: Commit staged changes
  Commit-->>Spark: Replaced Delta table

Loading

_{Reviews (4): Last reviewed commit: "Merge branch 'release/26.08' into rtas_d..." | Re-trigger Greptile}

[nartal1]

build

[nartal1]

build

[nartal1]

build

[firestarman]

lgtm