Accelerate DeltaDynamicPartitionOverwriteCommand

[jihoonson]

Fixes #13110.

Description

The insert overwrite queries with the dynamic partitionOverwriteMode to clustered tables currently fall back to CPU because of the lack of the GPU version of DeltaDynamicPartitionOverwriteCommand. This PR adds the missing classes to support it.

One thing to note is that we already have test cases for this query, which are test_delta_insert_overwrite_dynamic_sql_liquid_clustering and test_delta_insert_overwrite_df_liquid_clustering in delta_lake_liquid_clustering_test.py. These tests run some insert overwrite query and then verify whether it runs on GPU. These tests have been passing the CI because of #13059 even though the write falls back. These tests should cover the change in this PR.

Here is performance test result.

I used the query below against the TPC-H lineitem table at sf=100.

insert overwrite lineitem_inserted select * from parquet.`/parquet/lineitem` where mod(l_suppkey, 5) = 0;

I ran the query above on CPU and GPU, and compared their results. The means in the below snippet are the CPU runtime and the GPU runtime, respectively.

Means = 28439.0, 15296.0
Time diff = 13143.0
Speedup = 1.8592442468619246
T-Test (test statistic, p value, df) = 4.3441882924511575, 0.012212682511672469, 4.0
T-Test Confidence Interval = 4743.083707001302, 21542.9162929987
ALERT: significant change has been detected (p-value < 0.05)

Configs used:

GPU configs

export SPARK_CONF=("--master" "local[16]"
                   "--conf" "spark.driver.maxResultSize=2GB"
                   "--conf" "spark.driver.memory=16G"
                   "--conf" "spark.sql.files.maxPartitionBytes=2gb"
                   "--conf" "spark.plugins=com.nvidia.spark.SQLPlugin"
                   "--conf" "spark.rapids.memory.host.spillStorageSize=16G"
                   "--conf" "spark.rapids.memory.pinnedPool.size=8g"
                   "--conf" "spark.rapids.sql.concurrentGpuTasks=6"
                   "--conf" "spark.sql.sources.partitionOverwriteMode=DYNAMIC"
                   "--packages" "io.delta:delta-spark_2.12:3.3.1"
                   "--conf" "spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension"
                   "--conf" "spark.sql.catalog.spark_catalog=org.apache.spark.sql.delta.catalog.DeltaCatalog"
                   "--conf" "spark.driver.extraClassPath=$SPARK_RAPIDS_PLUGIN_JAR"
                   "--conf" "spark.executor.extraClassPath=$SPARK_RAPIDS_PLUGIN_JAR")

CPU configs

export SPARK_CONF=("--master" "local[64]"
                   "--conf" "spark.driver.memory=16G"
                   "--conf" "spark.scheduler.minRegisteredResourcesRatio=1.0"
                   "--conf" "spark.sql.adaptive.enabled=true"
                   "--conf" "spark.sql.broadcastTimeout=1200"
                   "--conf" "spark.dynamicAllocation.enabled=false"
                   "--conf" "spark.sql.sources.partitionOverwriteMode=DYNAMIC"
                   "--packages" "io.delta:delta-spark_2.12:3.3.1"
                   "--conf" "spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension"
                   "--conf" "spark.sql.catalog.spark_catalog=org.apache.spark.sql.delta.catalog.DeltaCatalog")

Checklists

• This PR has added documentation for new or modified features or behaviors.
• This PR has added new tests or modified existing tests to cover new code paths.
  (Please explain in the PR description how the new code paths are tested, such as names of the new/existing tests that cover them.)
• Performance testing has been performed and its results are added in the PR description. Or, an issue has been filed with a link in the PR description.

[Copilot]

Pull Request Overview

This PR adds GPU acceleration support for Delta Lake's dynamic partition overwrite functionality by implementing the missing GpuDeltaDynamicPartitionOverwriteCommand class and related infrastructure. Previously, queries with dynamic partition overwrite mode on clustered tables would fall back to CPU execution.

• Adds GpuDeltaDynamicPartitionOverwriteCommand class to provide GPU-accelerated execution
• Implements DeltaDynamicPartitionOverwriteCommandMeta for GPU compatibility checks and conversion
• Registers the new command in the Delta provider for automatic GPU acceleration

Reviewed Changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated no comments.

File	Description
GpuDeltaDynamicPartitionOverwriteCommand.scala	New GPU implementation of dynamic partition overwrite command
DeltaDynamicPartitionOverwriteCommandMeta.scala	Metadata class for GPU conversion and compatibility validation
Delta33xProvider.scala	Registers the new command for GPU acceleration

---

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

[jihoonson]

build

[razajafri]

LGTM