Assert RapidsDeltaWrite in integration tests [databricks] by jihoonson · Pull Request #13586 · NVIDIA/cudf-spark

jihoonson · 2025-10-08T22:28:08Z

Description

In integration tests, we can detect plans executed on CPU (so-called CPU fallback) to catch unintended execution on CPU. Every operator is expected to run on GPU unless it is explicitly specified in the allow_non_gpu marker. This works well in most cases, but has limitations:

It can check only the operators that appear in the query plan.
It can check only the queries that are executed via the Catalyst optimizer.

Due to these limitations, we currently cannot validate the delta lake writes. The delta lake writes usually do not appear in the query plan. Also, they are oftentimes directly created and executed in various Delta commands without going through the optimizer. As such, we had missed bugs in the past that the write is performed on CPU for certain Delta commands.

To address this issue, this PR adds a new function assert_rapids_delta_write in delta_lake_utils.py. This function executes the given function and captures all plans executed during executing the function. Once the execution is finished, it iterates over all captured plans to find the RapidsDeltaWrite, which must exist if the write was performed on GPU. Since Delta commands usually execute more than one plans, assert_rapids_delta_write assumes the plan is valid if RapidsDeltaWrite is found in at least one captured plan.

To avoid invasive change, the assert_gpu_and_cpu_writes_are_equal_collect and assert_gpu_and_cpu_save_as_table_are_equal_collect functions are modified to detect whether the current running test is a delta test and call the assert_rapids_delta_write function. There are some custom assert functions as well in some command tests. Those custom functions are also modified to call assert_rapids_delta_write. I manually confirmed all delta tests run successfully with Spark 3.5.5. Also verified some non-delta tests passing (parquet_write_test.py) as well.

Checklists

This PR has added documentation for new or modified features or behaviors.
This PR has added new tests or modified existing tests to cover new code paths.
(Please explain in the PR description how the new code paths are tested, such as names of the new/existing tests that cover them.)
Performance testing has been performed and its results are added in the PR description. Or, an issue has been filed with a link in the PR description.

Copilot

Pull Request Overview

This PR adds assertion functionality to detect when Delta Lake writes are executed on GPU versus CPU in integration tests. The primary purpose is to catch unintended CPU fallback scenarios that cannot be detected through traditional query plan validation.

Key changes:

New assert_rapids_delta_write function that captures execution plans and validates presence of RapidsDeltaWrite
Automatic detection and application of Delta write validation for tests with @delta_lake marker
Updates to Delta test functions to use the new assertion mechanism for proper GPU validation

Reviewed Changes

Copilot reviewed 12 out of 12 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
`ExecutionPlanCaptureCallback.scala`	Adds `contains` method interface to check for specific class names in execution plans
`ShimmedExecutionPlanCaptureCallbackImpl.scala`	Implements the `contains` method and updates copyright year
`delta_lake_utils.py`	Implements `assert_rapids_delta_write` function with plan capture and validation logic
`asserts.py`	Modifies write assertion functions to automatically apply Delta validation for Delta tests
`conftest.py`	Adds global tracking of `@delta_lake` marker for current test
Various test files	Updates test functions to use new assertion mechanism and adds xfail markers

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

jihoonson · 2025-10-08T22:30:24Z

This change revealed that the insert overwrite support with dynamic partition overwrite (#13110) is missing not only for clustered tables but also for regular tables.

jihoonson · 2025-10-08T22:34:51Z

build

razajafri

Just some minor comments

jihoonson · 2025-10-09T17:26:22Z

2025-10-08T23:15:33.1688006Z [2025-10-08T23:06:37.626Z] [2025-10-08T23:06:03.667Z] Done starting cluster
2025-10-08T23:15:33.1689180Z [2025-10-08T23:06:37.626Z] [2025-10-08T23:06:03.676Z] 2025-10-08 23:06:03,003 > [Thread-6] ERROR org.apache.spark.sql.connect.service.SparkConnectService - Could not start **** Connect GRPC service
2025-10-08T23:15:33.1689811Z [2025-10-08T23:06:37.626Z] [2025-10-08T23:06:03.676Z] java.io.IOException: Failed to bind to address /:15002

The CI failure was due to the bind failure in starting the spark connect server. Not sure why it failed though. Re-running the CI in case it was some intermittent issue.

jihoonson · 2025-10-09T17:26:25Z

build

jihoonson · 2025-10-09T21:08:40Z

[2025-10-09T18:24:14.310Z] [2025-10-09T18:23:52.589Z] =========================== short test summary info ============================
[2025-10-09T18:24:14.310Z] [2025-10-09T18:23:52.589Z] FAILED ../../src/main/python/delta_lake_write_test.py::test_delta_atomic_replace_table_as_select[False][DATAGEN_SEED=1760033326, TZ=UTC, IGNORE_ORDER({'local': True}), ALLOW_NON_GPU(DataWritingCommandExec,WriteFilesExec,DeserializeToObjectExec,ShuffleExchangeExec,FileSourceScanExec,FilterExec,MapPartitionsExec,MapElementsExec,ObjectHashAggregateExec,ProjectExec,SerializeFromObjectExec,SortExec)] - AssertionError: RapidsDeltaWrite is not found in any captured plan
[2025-10-09T18:24:14.310Z] [2025-10-09T18:23:52.589Z] FAILED ../../src/main/python/delta_lake_write_test.py::test_delta_atomic_create_table_as_select[False][DATAGEN_SEED=1760033326, TZ=UTC, IGNORE_ORDER({'local': True}), ALLOW_NON_GPU(DataWritingCommandExec,WriteFilesExec,DeserializeToObjectExec,ShuffleExchangeExec,FileSourceScanExec,FilterExec,MapPartitionsExec,MapElementsExec,ObjectHashAggregateExec,ProjectExec,SerializeFromObjectExec,SortExec)] - AssertionError: RapidsDeltaWrite is not found in any captured plan
[2025-10-09T18:24:14.310Z] [2025-10-09T18:23:52.589Z] FAILED ../../src/main/python/delta_lake_write_test.py::test_delta_rtas_sql[True-False][DATAGEN_SEED=1760033326, TZ=UTC, INJECT_OOM, IGNORE_ORDER({'local': True}), ALLOW_NON_GPU(DataWritingCommandExec,WriteFilesExec,DeserializeToObjectExec,ShuffleExchangeExec,FileSourceScanExec,FilterExec,MapPartitionsExec,MapElementsExec,ObjectHashAggregateExec,ProjectExec,SerializeFromObjectExec,SortExec)] - AssertionError: RapidsDeltaWrite is not found in any captured plan
[2025-10-09T18:24:14.310Z] [2025-10-09T18:23:52.589Z] FAILED ../../src/main/python/delta_lake_write_test.py::test_delta_ctas_sql[True-False][DATAGEN_SEED=1760033326, TZ=UTC, IGNORE_ORDER({'local': True}), ALLOW_NON_GPU(DataWritingCommandExec,WriteFilesExec,DeserializeToObjectExec,ShuffleExchangeExec,FileSourceScanExec,FilterExec,MapPartitionsExec,MapElementsExec,ObjectHashAggregateExec,ProjectExec,SerializeFromObjectExec,SortExec)] - AssertionError: RapidsDeltaWrite is not found in any captured plan
[2025-10-09T18:24:14.310Z] [2025-10-09T18:23:52.589Z] FAILED ../../src/main/python/delta_lake_write_test.py::test_delta_ctas_sql[False-False][DATAGEN_SEED=1760033326, TZ=UTC, IGNORE_ORDER({'local': True}), ALLOW_NON_GPU(DataWritingCommandExec,WriteFilesExec,DeserializeToObjectExec,ShuffleExchangeExec,FileSourceScanExec,FilterExec,MapPartitionsExec,MapElementsExec,ObjectHashAggregateExec,ProjectExec,SerializeFromObjectExec,SortExec)] - AssertionError: RapidsDeltaWrite is not found in any captured plan
[2025-10-09T18:24:14.310Z] [2025-10-09T18:23:52.589Z] FAILED ../../src/main/python/delta_lake_write_test.py::test_delta_rtas_sql[False-False][DATAGEN_SEED=1760033326, TZ=UTC, INJECT_OOM, IGNORE_ORDER({'local': True}), ALLOW_NON_GPU(DataWritingCommandExec,WriteFilesExec,DeserializeToObjectExec,ShuffleExchangeExec,FileSourceScanExec,FilterExec,MapPartitionsExec,MapElementsExec,ObjectHashAggregateExec,ProjectExec,SerializeFromObjectExec,SortExec)] - AssertionError: RapidsDeltaWrite is not found in any captured plan
[2025-10-09T18:24:14.310Z] [2025-10-09T18:23:52.589Z] = 6 failed, 289 passed, 101 skipped, 108 xfailed, 60 xpassed, 112 warnings in 900.29s (0:15:00) =

Some tests failed on databricks. Looking into it.

jihoonson · 2025-10-09T23:32:56Z

The test failures are because of #11169. I xfailed them.

jihoonson · 2025-10-10T00:32:26Z

build

jihoonson · 2025-10-14T16:41:47Z

build

razajafri · 2025-10-15T00:08:11Z

CI is failing on Databricks


[2025-10-14T17:03:44.330Z] [ERROR] [Error] /home/ubuntu/spark-rapids/sql-plugin/src/main/spark330db/scala/org/apache/spark/sql/rapids/execution/GpuBroadcastHashJoinExec.scala:179: type mismatch;

[2025-10-14T17:03:44.330Z]  found   : String("executor broadcast join stream")

[2025-10-14T17:03:44.330Z]  required: com.nvidia.spark.rapids.NvtxId

[2025-10-14T17:03:44.330Z] Error occurred in an application involving default arguments.

[2025-10-14T17:03:44.330Z] [ERROR] [Error] /home/ubuntu/spark-rapids/sql-plugin/src/main/spark330db/scala/org/apache/spark/sql/rapids/execution/GpuBroadcastHashJoinExec.scala:182: type mismatch;

[2025-10-14T17:03:44.330Z]  found   : Any

[2025-10-14T17:03:44.330Z]  required: org.apache.spark.sql.vectorized.ColumnarBatch

[2025-10-14T17:03:44.330Z] Error occurred in an application involving default arguments.

[2025-10-14T17:03:44.330Z] [ERROR] [Error] /home/ubuntu/spark-rapids/sql-plugin/src/main/spark330db/scala/org/apache/spark/sql/rapids/execution/GpuBroadcastHashJoinExec.scala:182: type mismatch;

[2025-10-14T17:03:44.330Z]  found   : Any

[2025-10-14T17:03:44.330Z]  required: Iterator[org.apache.spark.sql.vectorized.ColumnarBatch]

[2025-10-14T17:03:44.331Z] Error occurred in an application involving default arguments.

[2025-10-14T17:03:44.331Z] [ERROR] three errors found

You can use the jobs-for-developers to test all Databricks instances against your branch at the same time before re-running the CI, it might save us time

Signed-off-by: Jihoon Son <ghoonson@gmail.com>

jihoonson · 2025-10-15T20:21:26Z

build

jihoonson · 2025-10-16T00:27:34Z

This is interesting. The Ci has passed now without any fix for the previous build failure. I suspect some intermittent issue. I pushed one more commit to remove xfail marks for #13110 which has been resolved now. Also I rebased my branch and force pushed because I messed up my branch while merging the upstream. Apology for any inconvenience in the review.

jihoonson · 2025-10-16T00:29:28Z

You can use the jobs-for-developers to test all Databricks instances against your branch at the same time before re-running the CI, it might save us time

I also used an Nvidia's internal jenkins job to run the build and the integration tests on databricks 12.2, 13.3. and 14.3 as @razajafri suggested above.

gerashegalov

LGTM

I think we might have an issue in general that we don't register sub-executions for some Delta operations which makes it hard to navigate to underlying plans

razajafri

LGTM

jihoonson · 2025-10-17T20:34:34Z

I think we might have an issue in general that we don't register sub-executions for some Delta operations which makes it hard to navigate to underlying plans

Yeah, maybe we should extend this approach later as a follow-up to always capture all executions for Delta tests and verify whether they are on GPU.

Copilot AI review requested due to automatic review settings October 8, 2025 22:28

Copilot AI reviewed Oct 8, 2025

View reviewed changes

razajafri reviewed Oct 8, 2025

View reviewed changes

Comment thread integration_tests/src/main/python/delta_lake_delete_test.py Outdated

Comment thread integration_tests/src/main/python/asserts.py Outdated

sameerz added the test Only impacts tests label Oct 9, 2025

jihoonson added 9 commits October 15, 2025 10:36

fix auto compact, delete, liquid clustering tests

d3e4627

fix all except for merge test

b3c1cb5

fix merge test

ddf2645

use contains; misc cleanup

e015bd5

save as table

fe355cc

Signed-off-by: Jihoon Son <ghoonson@gmail.com>

copilot comments

587a072

xfail tests impacted by NVIDIA#11169

5036962

address comments

560a529

remove xfail for 13110

c5a8e21

jihoonson force-pushed the fix-13059 branch from f05e38e to c5a8e21 Compare October 15, 2025 17:47

gerashegalov approved these changes Oct 17, 2025

View reviewed changes

razajafri reviewed Oct 17, 2025

View reviewed changes

jihoonson merged commit f7ee922 into NVIDIA:branch-25.12 Oct 17, 2025
60 checks passed

gerashegalov mentioned this pull request Oct 21, 2025

[BUG] test_delta_overwrite_dynamic_missing_clauses fails with "RapidsDeltaWrite is not found in any captured plan" #13641

Closed

jihoonson mentioned this pull request Feb 19, 2026

Stronger typing for GpuV1FallbackWriters [databricks] #13426

Closed

3 tasks

Uh oh!

Conversation

jihoonson commented Oct 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Checklists

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jihoonson commented Oct 8, 2025

Uh oh!

jihoonson commented Oct 8, 2025

Uh oh!

razajafri left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

jihoonson commented Oct 9, 2025

Uh oh!

jihoonson commented Oct 9, 2025

Uh oh!

jihoonson commented Oct 9, 2025

Uh oh!

jihoonson commented Oct 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jihoonson commented Oct 10, 2025

Uh oh!

jihoonson commented Oct 14, 2025

Uh oh!

razajafri commented Oct 15, 2025

Uh oh!

jihoonson commented Oct 15, 2025

Uh oh!

jihoonson commented Oct 16, 2025

Uh oh!

jihoonson commented Oct 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gerashegalov left a comment

Choose a reason for hiding this comment

Uh oh!

razajafri left a comment

Choose a reason for hiding this comment

Uh oh!

jihoonson commented Oct 17, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

jihoonson commented Oct 8, 2025 •

edited

Loading

jihoonson commented Oct 9, 2025 •

edited

Loading

jihoonson commented Oct 16, 2025 •

edited

Loading