Skip to content

Add insert overwrite dynamic support for iceberg. - #13606

Merged
liurenjie1024 merged 6 commits into
NVIDIA:branch-25.12from
liurenjie1024:ray/13604
Oct 16, 2025
Merged

Add insert overwrite dynamic support for iceberg.#13606
liurenjie1024 merged 6 commits into
NVIDIA:branch-25.12from
liurenjie1024:ray/13604

Conversation

@liurenjie1024

Copy link
Copy Markdown
Collaborator

Fixes #13604.

Description

Add insert overwrite dynamic for iceberg.

Checklists

  • This PR has added documentation for new or modified features or behaviors.
  • This PR has added new tests or modified existing tests to cover new code paths.
    (Please explain in the PR description how the new code paths are tested, such as names of the new/existing tests that cover them.)
  • Performance testing has been performed and its results are added in the PR description. Or, an issue has been filed with a link in the PR description.

Copilot AI review requested due to automatic review settings October 14, 2025 07:32
Signed-off-by: liurenjie1024 <liurenjie2008@gmail.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR adds GPU support for INSERT OVERWRITE with dynamic partition mode for Iceberg tables, addressing issue #13604. The implementation allows dynamic partition overwrite operations to run on GPU when supported data types and partition functions are used.

  • Adds OverwritePartitionsDynamicExec execution plan support for GPU acceleration
  • Implements GpuOverwritePartitionsDynamicExec and GpuDynamicOverwrite classes to handle dynamic overwrite operations
  • Provides comprehensive test coverage with fallback scenarios for unsupported configurations

Reviewed Changes

Copilot reviewed 11 out of 11 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
tools/generated_files/354/supportedExecs.csv Adds OverwritePartitionsDynamicExec to supported operations list
tools/generated_files/354/operatorsScore.csv Assigns performance score of 3.0 to OverwritePartitionsDynamicExec
sql-plugin/src/main/spark350/scala/org/apache/spark/sql/execution/datasources/v2/WriteToDataSourceV2Exec.scala Implements GpuOverwritePartitionsDynamicExec execution plan
sql-plugin/src/main/spark350/scala/com/nvidia/spark/rapids/shims/Spark350PlusNonDBShims.scala Registers OverwritePartitionsDynamicExec with GPU execution rules
sql-plugin/src/main/scala/org/apache/spark/sql/rapids/ExternalSource.scala Adds tagging and conversion methods for OverwritePartitionsDynamicExec
sql-plugin/src/main/scala/com/nvidia/spark/rapids/v2WriteCommandMetas.scala Implements OverwritePartitionsDynamicExecMeta for GPU conversion
sql-plugin/src/main/scala/com/nvidia/spark/rapids/iceberg/IcebergProvider.scala Adds interface methods for dynamic overwrite support
integration_tests/src/main/python/iceberg/iceberg_insert_overwrite_dynamic_test.py Comprehensive test suite for dynamic overwrite functionality
iceberg/src/main/scala/org/apache/iceberg/spark/source/write.scala Implements GpuDynamicOverwrite class for Iceberg operations
iceberg/src/main/scala/org/apache/iceberg/spark/source/GpuSparkWrite.scala Updates toBatch method to handle dynamic overwrite operations
iceberg/src/main/scala/com/nvidia/spark/rapids/iceberg/IcebergProviderImpl.scala Implements GPU tagging and conversion for dynamic overwrite operations

Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.

Comment thread iceberg/src/main/scala/org/apache/iceberg/spark/source/write.scala
res-life
res-life previously approved these changes Oct 16, 2025

@res-life res-life left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@liurenjie1024

Copy link
Copy Markdown
Collaborator Author

build

@liurenjie1024

Copy link
Copy Markdown
Collaborator Author

build

@liurenjie1024
liurenjie1024 merged commit d1dc97b into NVIDIA:branch-25.12 Oct 16, 2025
60 checks passed
@liurenjie1024
liurenjie1024 deleted the ray/13604 branch October 16, 2025 12:02
@sameerz sameerz added the feature request New feature or request label Oct 16, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

feature request New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[FEA] Support insert into dynamic overwrite for iceberg.

6 participants