Skip to content

Add insert static overwrite support for iceberg. - #13611

Merged
liurenjie1024 merged 7 commits into
NVIDIA:branch-25.12from
liurenjie1024:ray/13605
Oct 17, 2025
Merged

Add insert static overwrite support for iceberg.#13611
liurenjie1024 merged 7 commits into
NVIDIA:branch-25.12from
liurenjie1024:ray/13605

Conversation

@liurenjie1024

Copy link
Copy Markdown
Collaborator

Fixes #13605 .

Description

Add insert static overwrite support for iceberg.

Checklists

  • This PR has added documentation for new or modified features or behaviors.
  • This PR has added new tests or modified existing tests to cover new code paths.
    (Please explain in the PR description how the new code paths are tested, such as names of the new/existing tests that cover them.)
  • Performance testing has been performed and its results are added in the PR description. Or, an issue has been filed with a link in the PR description.

Signed-off-by: liurenjie1024 <liurenjie2008@gmail.com>
Copilot AI review requested due to automatic review settings October 15, 2025 06:33

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR adds GPU support for Iceberg INSERT OVERWRITE (static overwrite) operations by implementing the OverwriteByExpressionExec execution plan on GPU. The implementation follows a similar pattern to the existing AppendDataExec GPU support.

Key changes:

  • Adds GPU execution plan class GpuOverwriteByExpressionExec for overwrite operations
  • Implements Iceberg-specific GPU write support for overwrite with GpuOverwriteByFilter class
  • Adds comprehensive integration tests covering various scenarios and fallback cases

Reviewed Changes

Copilot reviewed 11 out of 11 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
tools/generated_files/354/supportedExecs.csv Adds OverwriteByExpressionExec to supported GPU operations list
tools/generated_files/354/operatorsScore.csv Adds scoring entry for OverwriteByExpressionExec
sql-plugin/src/main/spark350/scala/org/apache/spark/sql/execution/datasources/v2/WriteToDataSourceV2Exec.scala Implements GpuOverwriteByExpressionExec GPU execution plan class
sql-plugin/src/main/spark350/scala/com/nvidia/spark/rapids/shims/Spark350PlusNonDBShims.scala Registers the new execution plan in the GPU conversion rules
sql-plugin/src/main/scala/org/apache/spark/sql/rapids/ExternalSource.scala Adds external source support for overwrite operations
sql-plugin/src/main/scala/com/nvidia/spark/rapids/v2WriteCommandMetas.scala Implements metadata handling for overwrite operations
sql-plugin/src/main/scala/com/nvidia/spark/rapids/iceberg/IcebergProvider.scala Extends provider interface for overwrite support
integration_tests/src/main/python/iceberg/iceberg_overwrite_static_test.py Comprehensive test suite for overwrite functionality
iceberg/src/main/scala/org/apache/iceberg/spark/source/write.scala Implements GpuOverwriteByFilter batch write class
iceberg/src/main/scala/org/apache/iceberg/spark/source/GpuSparkWrite.scala Updates GPU write logic to handle overwrite operations
iceberg/src/main/scala/com/nvidia/spark/rapids/iceberg/IcebergProviderImpl.scala Implements Iceberg-specific overwrite GPU conversion logic

Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.

Comment thread iceberg/src/main/scala/org/apache/iceberg/spark/source/GpuSparkWrite.scala Outdated
Comment thread iceberg/src/main/scala/org/apache/iceberg/spark/source/GpuSparkWrite.scala Outdated
Comment thread iceberg/src/main/scala/org/apache/iceberg/spark/source/GpuSparkWrite.scala Outdated

override def tagForGpu(cpuExec: OverwriteByExpressionExec,
meta: OverwriteByExpressionExecMeta): Unit = {
if (!meta.conf.isIcebergEnabled) {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NIT: extract a method?

res-life
res-life previously approved these changes Oct 16, 2025

@res-life res-life left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@liurenjie1024

Copy link
Copy Markdown
Collaborator Author

build

@sameerz sameerz added the feature request New feature or request label Oct 16, 2025
@liurenjie1024
liurenjie1024 merged commit 0f654ec into NVIDIA:branch-25.12 Oct 17, 2025
60 checks passed
@liurenjie1024
liurenjie1024 deleted the ray/13605 branch October 17, 2025 00:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

feature request New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[FEA] Support insert into overwrite static for iceberg.

5 participants