Add DBR 17.3 Delta CTAS/RTAS support and fix optimized writes [databricks] by nartal1 · Pull Request #15320 · NVIDIA/cudf-spark

nartal1 · 2026-07-22T05:31:30Z

Description

The initial Delta Lake write support for Databricks Runtime 17.3 kept atomic CTAS and RTAS operations on CPU. In addition, DBR 17.3 optimized writes could lose their target-partitioning marker during CPU-to-GPU shuffle conversion, preventing the optimized-write exchange from being reconstructed and resized correctly on GPU.

This PR enables the supported DBR 17.3 Delta CTAS/RTAS paths on GPU and fixes the optimized-write shuffle conversion.

In this PR:

Converts supported AtomicCreateTableAsSelectExec and AtomicReplaceTableAsSelectExec plans to the corresponding GPU atomic wrappers.
Retains DBR’s native catalog, staged-table, commit, abort, cache invalidation, and table-replacement semantics.
Keeps the query and Delta data-file write path GPU-eligible inside the atomic operation.
Preserves DeltaOptimizedWritePartitioning as the target partitioning while advertising the derived physical partitioning as the shuffle output.
Preserves optimized-write marker-local configuration overrides during CPU-to-GPU conversion.
Keeps the target and physical partitioning contracts consistent through tree copies, explicit partition-count changes, and AQE reconstruction.
Adds structural plan validation based on the DELTA_OPTIMIZED_WRITE shuffle origin, preventing unrelated GPU exchanges from satisfying the optimized-write plan check.
Adds focused conversion and AQE coverage for the DBR 17.3 optimized-write shuffle.

Expected fallback behavior

The implementation continues to fall back to CPU for table-creation features whose DBR metadata or catalog semantics have not been qualified for the GPU create path, including:

Row filters and column masks.
Liquid clustering and auto-TTL.
Catalog-owned tables and coordinated commits.
Explicit or session-default deletion-vector creation settings that require unsupported metadata handling.

When Delta optimized writes are enabled with AQE, the DELTA_OPTIMIZED_WRITE exchange can run on GPU. With optimized writes enabled and AQE disabled, DBR uses its non-AQE DeltaOptimizedWriterExec; that path remains an expected CPU fallback.

Performance

The scaled workload used:

Databricks Runtime 17.3, Spark 4.0, and Scala 2.13.
Eight g4dn.4xlarge T4 workers and a matching driver.
A deterministic SF100-class Delta source with 452,433,852 rows across 12 source tables and 27.36 GB of compressed source data.
Fourteen representative model-build transformations producing 425,795,424 final rows.

This is an SF100-class workload and is not an official TPC-DS SF100 benchmark.

GPU Delta writer benefit

Delta optimized writes were disabled in this comparison so that it isolates the Delta writer change.

Configuration	Delta writer	Fresh CTAS	Replacement RTAS
RAPIDS 26.06 release	CPU fallback	138.819 s	133.876 s
26.08 SNAPSHOT CPU-writer control	CPU	128.665 s	126.982 s
26.08 SNAPSHOT with this GPU write path	GPU	92.874 s	90.778 s

Compared with the same-26.08 CPU-writer control, GPU Delta writing was:

1.385x faster for fresh table creation.
1.399x faster for replacement.

Compared with the RAPIDS 26.06 released behavior, it was:

1.495x faster for fresh table creation, a 33.1% wall-time reduction.
1.475x faster for replacement, a 32.2% wall-time reduction.

Physical-plan validation confirmed that the 26.06 run accelerated the query operators but crossed through GpuColumnarToRow into CPU WriteIntoDeltaCommand/WriteFiles. The 26.08 GPU result used the GPU atomic wrapper and GPU Delta writer.

Delta optimized-write tradeoff

spark.databricks.delta.optimizeWrite.enabled=false/true using the build jar from this PR.

Delta optimized writes	Fresh median	Replacement median	Output files
Off	95.724 s	95.318 s	2,004
On, with AQE	117.304 s	116.041 s	84

Enabling optimized writes increased write time by 22.5% for fresh creation and 21.7% for replacement. In exchange, it reduced the output from 2,004 files to 84 files, a 23.86x reduction.

This was not caused by CPU fallback: the optimize-on runs used GPU atomic wrappers, GPU Delta writing, and GPU DELTA_OPTIMIZED_WRITE exchanges. The additional time represents the actual repartitioning and file-sizing work required to produce the more compact layout.

Checklists

Documentation

Updated for new or modified user-facing features or behaviors
No user-facing change

Testing

Added or modified tests to cover new code paths
Covered by existing tests
(Please provide the names of the existing tests in the PR description.)
Not required

Performance

Tests ran and results are added in the PR description
Issue filed with a link in the PR description
Not required

Signed-off-by: Niranjan Artal <nartal@nvidia.com>

…lta-write

Signed-off-by: Niranjan Artal <nartal@nvidia.com>

greptile-apps · 2026-07-22T05:48:26Z

Greptile Summary

This PR enables GPU acceleration for DBR 17.3 AtomicCreateTableAsSelectExec and AtomicReplaceTableAsSelectExec (CTAS/RTAS), and fixes a bug where the DeltaOptimizedWritePartitioning marker was silently dropped during CPU-to-GPU shuffle conversion, preventing DELTA_OPTIMIZED_WRITE exchanges from being correctly resized and reconstructed by AQE.

CTAS/RTAS GPU wrappers: New GpuAtomicCreateTableAsSelectExec and GpuAtomicReplaceTableAsSelectExec for 400db173 retain DBR's native catalog, staged-table, commit, and abort semantics, while allowing the nested WriteIntoDeltaCommand to run on GPU via a GpuAtomicDeltaWriteContext Spark local-property guard set at execution time so the replanning of the nested write sees the GPU-active context.
Optimized-write shuffle fix: GpuShuffleExchangeExec now stores the full DeltaOptimizedWritePartitioning as targetOutputPartitioning, resolves the physical partition count from createDynamicPhysicalPartitioning(inputNumPartitions) at shuffle-dependency-build time, and uses the advertised physical HashPartitioning (not the DBR marker) as the basis for withNewNumPartitions/repartition so AQE resizing stays consistent.
Scope guards: GpuWriteIntoDeltaCommand is now gated to the atomic write stack via GpuAtomicDeltaWriteContext.isActive, and a new GpuLiquidOptimizeWriteIntoDeltaCommand is introduced for the liquid-OPTIMIZE call stack, with the provider dispatching between them at plan time.

Confidence Score: 5/5

Safe to merge; the DBR 17.3 atomicity, catalog-commit, and partitioning contracts are all preserved and guarded by purpose-built context sentinels.

The CTAS/RTAS wrappers correctly delegate staging, commit, and abort to DBR's native catalog while enabling GPU execution only inside the GpuAtomicDeltaWriteContext scope. The optimized-write shuffle fix accurately replicates DBR's two-view partitioning contract (marker as target, physical hash as output) and resolves the dynamic partition count at the right point in the execution lifecycle. Shim splits, resource management, and context propagation all look correct.

Files Needing Attention: No files require special attention.

Important Files Changed

Filename	Overview
sql-plugin/src/main/spark400db173/scala/org/apache/spark/rapids/shims/GpuShuffleExchangeExec.scala	Significant rework to preserve DeltaOptimizedWritePartitioning as targetOutputPartitioning and resolve physical partition count at shuffle-dependency-build time; withNewNumPartitions/repartition now correctly derive the new target from the advertised physical partitioning
sql-plugin/src/main/spark400db173/scala/org/apache/spark/sql/execution/datasources/v2/rapids/GpuAtomicCreateTableAsSelectExec.scala	New GPU wrapper for DBR 17.3 atomic CTAS; row-based execution delegates to writeToTable inside GpuAtomicDeltaWriteContext.withAtomicWrite so nested write planning sees the active GPU context
sql-plugin/src/main/spark400db173/scala/org/apache/spark/sql/execution/datasources/v2/rapids/GpuAtomicReplaceTableAsSelectExec.scala	New GPU wrapper for DBR 17.3 atomic RTAS; mirrors CTAS pattern with stageCreateOrReplace/stageReplace logic and GpuAtomicDeltaWriteContext guard
sql-plugin/src/main/spark400db173/scala/org/apache/spark/sql/execution/datasources/v2/rapids/GpuAtomicDeltaWriteContext.scala	Thread-safe UUID-based context guard for nested atomic Delta writes; uses Spark local properties with save/restore semantics for re-entrancy
delta-lake/delta-spark400db173/src/main/scala/com/nvidia/spark/rapids/delta/DeltaSpark400DB173Provider.scala	Removes the forced CPU fallback for CTAS/RTAS and adds proper convertToGpu implementations; dispatches WriteIntoDeltaCommand meta creation between GpuLiquidOptimizeWriteIntoDeltaCommandMeta and GpuWriteIntoDeltaCommandMeta based on runtime context
delta-lake/delta-spark400db173/src/main/scala/com/databricks/sql/transaction/tahoe/rapids/GpuLiquidOptimizeWriteIntoDeltaCommand.scala	New GPU write command for the native liquid OPTIMIZE call stack; includes stats schema extraction refactored from GpuWriteIntoDeltaCommand
integration_tests/src/main/python/delta_lake_write_test.py	Adds comprehensive DBR 17.3 CTAS/RTAS GPU test coverage: managed and SQL paths, optimized-write with/without AQE, IF NOT EXISTS / REPLACE branches, empty-input, legacy non-AQE DeltaOptimizedWriterExec
integration_tests/src/main/python/delta_lake_utils.py	Adds assert_db173_gpu_data_writing_command helper with GPU plan capture and structured validation of atomic wrappers, DELTA_OPTIMIZED_WRITE exchange origin, AQE final-plan stage, and absence of CPU fallback; also adds assert_delta_add_file_stats_present for AddFile statistics integrity

_{Reviews (11): Last reviewed commit: "address review comments" | Re-trigger Greptile}

nartal1 · 2026-07-22T05:51:06Z

build

Signed-off-by: Niranjan Artal <nartal@nvidia.com>

nartal1 · 2026-07-22T19:12:26Z

build

Signed-off-by: Niranjan Artal <nartal@nvidia.com>

nartal1 · 2026-07-22T20:25:44Z

build

nartal1 · 2026-07-22T21:20:03Z

build

nartal1 · 2026-07-23T03:18:01Z

build

Signed-off-by: Niranjan Artal <nartal@nvidia.com>

nartal1 · 2026-07-23T18:57:23Z

build

Signed-off-by: Niranjan Artal <nartal@nvidia.com>

nartal1 · 2026-07-23T23:20:49Z

build

firestarman

lgtm

firestarman · 2026-07-27T06:25:45Z

+        assert effective_optimize_write.lower() == str(optimized_write).lower()
+        effective_aqe = spark.conf.get("spark.sql.adaptive.enabled")
+        assert effective_aqe.lower() == str(aqe_enabled).lower()
+        command = "CREATE OR REPLACE TABLE" if replace else "CREATE TABLE"


nit: Could we add coverage for the existing-table branches of the new atomic CTAS/RTAS wrappers? In particular, CREATE TABLE IF NOT EXISTS, the existing-table error for plain CREATE, and REPLACE TABLE without OR CREATE.

firestarman · 2026-07-27T06:25:45Z

+    source_table = spark_tmp_table_factory.get() + '_sql_source'
+
+    with_cpu_session(
+        lambda spark: spark.range(81920).selectExpr(


nit: Could we add a focused zero-row CTAS/RTAS case? The new tests only exercise non-empty inputs, so the schema-only/empty commit path is not covered.

firestarman · 2026-07-27T06:25:45Z

+/**
+ * GPU wrapper for DBR 17.3 atomic RTAS.
+ *
+ * DBR's native catalog remains responsible for stage/commit/abort and therefore for Unity


nit: Could we clarify that only the native stage/commit/abort mechanics are retained here, while catalog-owned and coordinated-commit table creation remains blocked pending GPU qualification? The current wording can read as though those semantics are supported.

firestarman · 2026-07-27T06:25:45Z

+  override protected def run(): Seq[InternalRow] = {
+    if (catalog.tableExists(ident)) {
+      if (ifNotExists) {
+        return Nil


nit: Could this avoid explicit returns and use expression-style branching or matching instead? The repository’s Scala convention prefers the final expression as the result.

firestarman · 2026-07-27T06:25:45Z

+  private val activeKey = "spark.rapids.sql.delta.atomicWrite.active"
+
+  def isActive: Boolean = SparkContext.getActive
+    .exists(_.getLocalProperty(activeKey) == "true")


nit: Could this context use a private per-JVM token rather than the public literal "true"? Application code can set Spark local properties, so a fixed value makes the qualification gate externally reproducible.

nvauto · 2026-07-27T06:53:25Z

NOTE: release/26.08 has been created from main. Please retarget your PR to release/26.08 if it should be included in the release.

The base branch was changed.

Signed-off-by: Niranjan Artal <nartal@nvidia.com>

nartal1 · 2026-07-27T19:57:47Z

build

firestarman

LGTM

Closes #15059. ### Description This PR updates the download and archive documentation for the v26.08.0 release, adds the Databricks compatibility guidance requested in #15059 directly to the download page, and aligns the Code of Conduct contact with the cuDF Spark project name. Changes include: - Update `docs/download.md` to list v26.08.0 artifacts, Maven coordinates, signature verification commands, supported Spark versions, and concise release notes. - Add an inline Databricks compatibility section to `docs/download.md`, covering the current DBR 14.3/17.3 Spark, Scala, JDK, CUDA, driver, Delta feature, and fallback guidance without introducing a separate support page. - Update DBR 17.3 coverage for v26.08 liquid clustering and CTAS/RTAS support based on #15278 and #15320. - Move the previous current release, v26.06.1, into `docs/archive.md` as the newest archived release. - Replace `spark-rapids-conduct@nvidia.com` with `cudf-spark-conduct@nvidia.com` in `CODE_OF_CONDUCT.md`. - Verify the Spark support matrix against `pom.xml` on `release/26.08`. - Verify every PR referenced by the release notes is included in `release/26.08`. - Verify the archived v26.06.1 requirements and artifacts against the `v26.06.1` tag. Validation: - `git diff --check -- CODE_OF_CONDUCT.md docs/download.md docs/archive.md` - IDE lints for all three documentation files - Verified the inline Databricks matrix excludes the unsupported DBR 13.3 line ### Checklists Documentation - [x] Updated for new or modified user-facing features or behaviors - [ ] No user-facing change Testing - [ ] Added or modified tests to cover new code paths - [ ] Covered by existing tests (Please provide the names of the existing tests in the PR description.) - [x] Not required Performance - [ ] Tests ran and results are added in the PR description - [ ] Issue filed with a link in the PR description - [x] Not required --------- Signed-off-by: liyuan <yuali@nvidia.com>

nartal1 added 12 commits July 10, 2026 19:45

[WIP] Add DBR 17.3 native Delta write path

41af2e3

Signed-off-by: Niranjan Artal <nartal@nvidia.com>

[databricks] Complete DBR 17.3 native Delta writes

fbe80ad

Signed-off-by: Niranjan Artal <nartal@nvidia.com>

update tests

830660e

Signed-off-by: Niranjan Artal <nartal@nvidia.com>

[databricks] Validate DBR 17.3 native Delta writes

215a059

Signed-off-by: Niranjan Artal <nartal@nvidia.com>

Merge remote-tracking branch 'origin/main' into codex/db173-native-de…

303a996

…lta-write

[databricks] Record final Delta write validation

08e3c4e

Signed-off-by: Niranjan Artal <nartal@nvidia.com>

[databricks] Accelerate DBR 17.3 atomic Delta writes

52386bb

Signed-off-by: Niranjan Artal <nartal@nvidia.com>

[databricks] Fix optimized write shuffle partitioning

e9bacfc

Signed-off-by: Niranjan Artal <nartal@nvidia.com>

[databricks] Validate optimized write conversion and AQE

7f30e55

Signed-off-by: Niranjan Artal <nartal@nvidia.com>

[databricks] Qualify atomic optimized write paths

862b2bb

Signed-off-by: Niranjan Artal <nartal@nvidia.com>

[databricks] Match legacy optimized write structurally

bbf5c24

Signed-off-by: Niranjan Artal <nartal@nvidia.com>

[databricks] Remove development-only DBR artifacts

cc76309

Signed-off-by: Niranjan Artal <nartal@nvidia.com>

nartal1 self-assigned this Jul 22, 2026

nartal1 added the task Work required that improves the product but is not user facing label Jul 22, 2026

Merge origin/main into ctas-rtas-write-support

e072180

Signed-off-by: Niranjan Artal <nartal@nvidia.com>

greptile-apps Bot reviewed Jul 22, 2026

View reviewed changes

Comment thread ...ugin/src/main/spark400db173/scala/org/apache/spark/rapids/shims/GpuShuffleExchangeExec.scala

Comment thread integration_tests/src/main/python/delta_lake_write_test.py

nartal1 added 5 commits July 22, 2026 10:23

[databricks] Fix DBR 17.3 Delta write validation

a40d14a

Signed-off-by: Niranjan Artal <nartal@nvidia.com>

[databricks] Preserve nested Delta write fallback

06932b0

Signed-off-by: Niranjan Artal <nartal@nvidia.com>

[databricks] Validate retained Delta source options

dfe3b14

Signed-off-by: Niranjan Artal <nartal@nvidia.com>

[databricks] Defer Delta source write validation

7200dd9

Signed-off-by: Niranjan Artal <nartal@nvidia.com>

[databricks] Scope DBR 17.3 native Delta writes

7f5e830

Signed-off-by: Niranjan Artal <nartal@nvidia.com>

[databricks] Fix Scala import grouping

9c9789f

Signed-off-by: Niranjan Artal <nartal@nvidia.com>

nartal1 requested a review from a team July 23, 2026 00:42

Allow empty clustered Delta read plans

ebb7b99

Signed-off-by: Niranjan Artal <nartal@nvidia.com>

nartal1 and others added 2 commits July 23, 2026 11:52

Update clustered Delta read copyright

907b373

Signed-off-by: Niranjan Artal <nartal@nvidia.com>

Merge branch 'main' into ctas-rtas-write-support

d5cf57d

Merge remote-tracking branch 'origin/main' into ctas-rtas-write-support

0d71bf5

Signed-off-by: Niranjan Artal <nartal@nvidia.com>

firestarman reviewed Jul 27, 2026

View reviewed changes

firestarman previously approved these changes Jul 27, 2026

View reviewed changes

sameerz added the feature request New feature or request label Jul 27, 2026

nartal1 changed the base branch from main to release/26.08 July 27, 2026 17:49

nartal1 requested a review from firestarman July 27, 2026 17:50

nartal1 and others added 2 commits July 27, 2026 10:50

Merge branch 'release/26.08' into ctas-rtas-write-support

066ae24

address review comments

ec1d083

Signed-off-by: Niranjan Artal <nartal@nvidia.com>

firestarman approved these changes Jul 28, 2026

View reviewed changes

nartal1 merged commit 41129e5 into NVIDIA:release/26.08 Jul 28, 2026
55 checks passed

nvliyuan mentioned this pull request Aug 4, 2026

[DOC] update download page for 26.08 release [skip ci] #15509

Merged

8 tasks

Uh oh!

Conversation

nartal1 commented Jul 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Expected fallback behavior

Performance

GPU Delta writer benefit

Delta optimized-write tradeoff

Checklists

Uh oh!

greptile-apps Bot commented Jul 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Greptile Summary

Confidence Score: 5/5

Important Files Changed

Uh oh!

Uh oh!

Uh oh!

nartal1 commented Jul 22, 2026

Uh oh!

nartal1 commented Jul 22, 2026

Uh oh!

nartal1 commented Jul 22, 2026

Uh oh!

nartal1 commented Jul 22, 2026

Uh oh!

nartal1 commented Jul 23, 2026

Uh oh!

nartal1 commented Jul 23, 2026

Uh oh!

nartal1 commented Jul 23, 2026

Uh oh!

firestarman left a comment

Choose a reason for hiding this comment

Uh oh!

firestarman Jul 27, 2026

Choose a reason for hiding this comment

Uh oh!

firestarman Jul 27, 2026

Choose a reason for hiding this comment

Uh oh!

firestarman Jul 27, 2026

Choose a reason for hiding this comment

Uh oh!

firestarman Jul 27, 2026

Choose a reason for hiding this comment

Uh oh!

firestarman Jul 27, 2026

Choose a reason for hiding this comment

Uh oh!

nvauto commented Jul 27, 2026

Uh oh!

nartal1 commented Jul 27, 2026

Uh oh!

firestarman left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

nartal1 commented Jul 22, 2026 •

edited

Loading

greptile-apps Bot commented Jul 22, 2026 •

edited

Loading