Skip to content

Commit 4383de9

Browse files
firestarmanfirestarman
authored
Add Spark 4.0.4 shim support[databricks] (#15313)
Fixes #15270. ### Description - Add the Spark 4.0.4 Maven profile, shim service provider, version checks, generated support matrices, and shared shim annotations so the plugin can build and load against Spark 4.0.4. - Extend Spark-version helpers and AST compatibility handling for Spark 4.0.4 because ACOSH uses a CPU fallback path on this release. - Include Spark 4.0.4 in the download documentation so users can identify it as a supported Apache Spark version. - Validate the complete Scala 2.13 / CUDA 13 reactor with `SPARK_HOME=/bigdata/work/tools/spark-4.0.4-bin-hadoop3 mvn -s ~/.m2/settings_art.xml -f scala2.13/pom.xml -Dbuildver=404 -Dcuda.version=cuda13 -Dspark.test.home=\"$SPARK_HOME\" package`; all 15 modules passed, including 2,138 Scala tests with 0 failures. - Local NDS performance results with RAPIDS shuffle writer threads set to `8` show no overall performance regression observed for Spark 4.0.4. | Item | Value | | ------------- | ---------------------------------------------------------- | | Dataset | /bigdata/tpcds_data/parquet_100f | | Format | Parquet | | RAPIDS jar | rapids-4-spark_2.13-26.08.0-SNAPSHOT-cuda13-403-404.jar | | Spark 4.0.3 | spark-4.0.3-bin-hadoop3 | | Spark 4.0.4 | spark-4.0.4-bin-hadoop3 | | Runs | 3 per shim | | Compared runs | Run 1 and run 2 | | Shuffle conf | `spark.rapids.shuffle.multiThreaded.writer.threads=8` | | Query status | 618 query JSON records checked, 0 non-Completed | | Result | No overall performance regression observed for Spark 4.0.4 | | Metric | Spark 4.0.3 | Spark 4.0.4 | 4.0.4 vs 4.0.3 | | -------------------------- | ----------- | ----------- | -------------- | | Power run 1 | 770.000s | 744.000s | -3.38% | | Power run 2 | 750.000s | 752.000s | +0.27% | | Power run avg | 760.000s | 748.000s | -1.58% | | Sum of per-query avg times | 759.059s | 746.649s | -1.63% | ### Checklists Documentation - [x] Updated for new or modified user-facing features or behaviors - [ ] No user-facing change Testing - [ ] Added or modified tests to cover new code paths - [x] Covered by existing tests (Please provide the names of the existing tests in the PR description.) - [ ] Not required Performance - [x] Tests ran and results are added in the PR description - [ ] Issue filed with a link in the PR description - [ ] Not required Signed-off-by: Firestarman <firestarmanllc@gmail.com>
1 parent 9db460f commit 4383de9

214 files changed

Lines changed: 1828 additions & 12 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎datagen/src/main/spark400/scala/org/apache/spark/sql/tests/datagen/DataGenExprShims.scala‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -20,11 +20,13 @@
2020
{"spark": "401"}
2121
{"spark": "402"}
2222
{"spark": "403"}
23+
{"spark": "404"}
2324
{"spark": "411"}
2425
{"spark": "412"}
2526
{"spark": "413"}
2627
{"spark": "420"}
2728
spark-rapids-shim-json-lines ***/
29+
2830
package org.apache.spark.sql.tests.datagen
2931

3032
import org.apache.spark.sql.Column

‎docs/download.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -44,10 +44,10 @@ The plugin is designed to work on NVIDIA Volta, Turing, Ampere, Ada Lovelace, Ho
4444
Apache Spark 3.3.0, 3.3.1, 3.3.2, 3.3.3, 3.3.4
4545
Apache Spark 3.4.0, 3.4.1, 3.4.2, 3.4.3, 3.4.4
4646
Apache Spark 3.5.0, 3.5.1, 3.5.2, 3.5.3, 3.5.4, 3.5.5, 3.5.6, 3.5.7, 3.5.8
47-
Apache Spark 4.0.0, 4.0.1, 4.0.2, 4.0.3
47+
Apache Spark 4.0.0, 4.0.1, 4.0.2, 4.0.3, 4.0.4
4848
Apache Spark 4.1.1
4949
Scala 2.12: Spark 3.3.0 through 3.5.8
50-
Scala 2.13: Spark 3.5.0 through 3.5.8, and Spark 4.0.0, 4.0.1, 4.0.2, 4.0.3, and 4.1.1
50+
Scala 2.13: Spark 3.5.0 through 3.5.8, and Spark 4.0.0, 4.0.1, 4.0.2, 4.0.3, 4.0.4, and 4.1.1
5151

5252
Supported Databricks runtime versions for Azure and AWS:
5353
Databricks 13.3 ML LTS (GPU, Scala 2.12, Spark 3.4.1)

‎iceberg/iceberg-1-10-x/src/main/spark400/java/com/nvidia/spark/rapids/iceberg/GpuInternalRow.java‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,7 @@
1818
{"spark": "401"}
1919
{"spark": "402"}
2020
{"spark": "403"}
21+
{"spark": "404"}
2122
spark-rapids-shim-json-lines ***/
2223

2324
package com.nvidia.spark.rapids.iceberg;

‎iceberg/iceberg-1-11-x/src/main/spark400/java/com/nvidia/spark/rapids/iceberg/GpuInternalRow.java‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,7 @@
1818
{"spark": "401"}
1919
{"spark": "402"}
2020
{"spark": "403"}
21+
{"spark": "404"}
2122
spark-rapids-shim-json-lines ***/
2223

2324
package com.nvidia.spark.rapids.iceberg;

‎iceberg/iceberg-1-11-x/src/main/spark402/scala/com/nvidia/spark/rapids/iceberg/iceberg111x/IcebergProviderImpl.scala‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,7 @@
1616
/*** spark-rapids-shim-json-lines
1717
{"spark": "402"}
1818
{"spark": "403"}
19+
{"spark": "404"}
1920
spark-rapids-shim-json-lines ***/
2021

2122
package com.nvidia.spark.rapids.iceberg.iceberg111x

‎integration_tests/src/main/python/ast_test.py‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -17,7 +17,7 @@
1717
from asserts import assert_cpu_and_gpu_are_equal_collect_with_capture, assert_gpu_and_cpu_are_equal_collect
1818
from data_gen import *
1919
from marks import approximate_float, datagen_overrides, ignore_order, disable_ansi_mode
20-
from spark_session import with_cpu_session, is_spark_403, is_spark_412_or_later, \
20+
from spark_session import with_cpu_session, is_spark_403_or_404, is_spark_412_or_later, \
2121
is_spark_420_or_later
2222
import pyspark.sql.functions as f
2323

@@ -61,9 +61,9 @@
6161

6262
ast_boolean_descr = [(boolean_gen, True)]
6363
ast_double_descr = [(double_gen, True)]
64-
# AST is not expressive enough to support the ACOSH Spark emulation expression in Spark 4.0.3
65-
# and Spark 4.1.2+.
66-
ast_acosh_descr = [(double_gen, not (is_spark_403() or is_spark_412_or_later()))]
64+
# AST is not expressive enough to support the ACOSH Spark emulation expression in Spark 4.0.3,
65+
# Spark 4.0.4, and Spark 4.1.2+.
66+
ast_acosh_descr = [(double_gen, not (is_spark_403_or_404() or is_spark_412_or_later()))]
6767

6868
_project_ast_enabled_conf = {"spark.rapids.sql.projectAstEnabled": "true"}
6969

‎integration_tests/src/main/python/spark_session.py‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -267,6 +267,12 @@ def is_spark_401_or_later():
267267
def is_spark_403():
268268
return spark_version() == "4.0.3"
269269

270+
def is_spark_404():
271+
return spark_version() == "4.0.4"
272+
273+
def is_spark_403_or_404():
274+
return is_spark_403() or is_spark_404()
275+
270276
def is_spark_411_or_later():
271277
return spark_version() >= "4.1.1"
272278

‎integration_tests/src/test/spark400/scala/org/apache/spark/sql/rapids/shims/TrampolineUtilShim.scala‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -20,12 +20,14 @@
2020
{"spark": "401"}
2121
{"spark": "402"}
2222
{"spark": "403"}
23+
{"spark": "404"}
2324
{"spark": "411"}
2425
{"spark": "412"}
2526
{"spark": "413"}
2627
{"spark": "420"}
2728
spark-rapids-shim-json-lines ***/
2829

30+
2931
package org.apache.spark.sql.rapids.shims
3032

3133
import org.apache.spark.sql.classic.SparkSession

‎pom.xml‎

Lines changed: 30 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -704,6 +704,32 @@
704704
<module>iceberg/iceberg-1-11-x</module>
705705
</modules>
706706
</profile>
707+
<profile>
708+
<id>release404</id>
709+
<activation>
710+
<property>
711+
<name>buildver</name>
712+
<value>404</value>
713+
</property>
714+
</activation>
715+
<properties>
716+
<buildver>404</buildver>
717+
<spark.version>${spark404.version}</spark.version>
718+
<spark.test.version>${spark404.version}</spark.test.version>
719+
<parquet.hadoop.version>1.13.1</parquet.hadoop.version>
720+
<rapids.delta.artifactId1>rapids-4-spark-delta-40x</rapids.delta.artifactId1>
721+
<iceberg.artifact.suffix>${spark40x.iceberg.artifact.suffix}</iceberg.artifact.suffix>
722+
<iceberg.runtime.version>${iceberg.110x.version}</iceberg.runtime.version>
723+
<rapids.iceberg.artifactId>rapids-4-spark-iceberg-1-10-x</rapids.iceberg.artifactId>
724+
<rapids.iceberg.artifactId2>rapids-4-spark-iceberg-1-11-x</rapids.iceberg.artifactId2>
725+
<slf4j.version>2.0.7</slf4j.version>
726+
</properties>
727+
<modules>
728+
<module>delta-lake/delta-40x</module>
729+
<module>iceberg/iceberg-1-10-x</module>
730+
<module>iceberg/iceberg-1-11-x</module>
731+
</modules>
732+
</profile>
707733
<profile>
708734
<id>release411</id>
709735
<activation>
@@ -1035,6 +1061,7 @@
10351061
<spark401.version>4.0.1</spark401.version>
10361062
<spark402.version>4.0.2</spark402.version>
10371063
<spark403.version>4.0.3</spark403.version>
1064+
<spark404.version>4.0.4</spark404.version>
10381065
<spark411.version>4.1.1</spark411.version>
10391066
<spark412.version>4.1.2</spark412.version>
10401067
<spark413.version>4.1.3</spark413.version>
@@ -1755,14 +1782,14 @@ This will force full Scala code rebuild in downstream modules.
17551782
<requireProperty>
17561783
<property>buildver</property>
17571784
<regex>^(33[0-4]|34[0-4]|35[0-8]|341db|350db143)$</regex>
1758-
<regexMessage>Unsupported buildver value ${buildver} for a Scala 2.12 build. Supported Scala 2.12 buildver values are: 330, 331, 332, 333, 334, 340, 341, 342, 343, 344, 350, 351, 352, 353, 354, 355, 356, 357, 358, 341db, 350db143. For Scala 2.13/Spark 4.x builds, use -f scala2.13/pom.xml with a supported Scala 2.13 buildver such as 350, 400, 403, 412, 413, or 420.</regexMessage>
1785+
<regexMessage>Unsupported buildver value ${buildver} for a Scala 2.12 build. Supported Scala 2.12 buildver values are: 330, 331, 332, 333, 334, 340, 341, 342, 343, 344, 350, 351, 352, 353, 354, 355, 356, 357, 358, 341db, 350db143. For Scala 2.13/Spark 4.x builds, use -f scala2.13/pom.xml with a supported Scala 2.13 buildver such as 350, 400, 403, 404, 412, 413, or 420.</regexMessage>
17591786
</requireProperty>
17601787
<!-- #endif scala-2.12 -->
17611788
<!-- #if scala-2.13 --><!--
17621789
<requireProperty>
17631790
<property>buildver</property>
1764-
<regex>^(35[0-8]|40[0-3]|41[1-3]|420|400db173)$</regex>
1765-
<regexMessage>Unsupported buildver value ${buildver} for a Scala 2.13 build. Supported Scala 2.13 buildver values are: 350, 351, 352, 353, 354, 355, 356, 357, 358, 400, 401, 402, 403, 411, 412, 413, 420, 400db173. Spark 3.3.x and 3.4.x builds use Scala 2.12; for example, build Spark 330 with the root pom.xml instead of scala2.13/pom.xml.</regexMessage>
1791+
<regex>^(35[0-8]|40[0-4]|41[1-3]|420|400db173)$</regex>
1792+
<regexMessage>Unsupported buildver value ${buildver} for a Scala 2.13 build. Supported Scala 2.13 buildver values are: 350, 351, 352, 353, 354, 355, 356, 357, 358, 400, 401, 402, 403, 404, 411, 412, 413, 420, 400db173. Spark 3.3.x and 3.4.x builds use Scala 2.12; for example, build Spark 330 with the root pom.xml instead of scala2.13/pom.xml.</regexMessage>
17661793
</requireProperty>
17671794
--><!-- #endif scala-2.13 -->
17681795
<requireProperty>

‎scala2.13/pom.xml‎

Lines changed: 30 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -704,6 +704,32 @@
704704
<module>iceberg/iceberg-1-11-x</module>
705705
</modules>
706706
</profile>
707+
<profile>
708+
<id>release404</id>
709+
<activation>
710+
<property>
711+
<name>buildver</name>
712+
<value>404</value>
713+
</property>
714+
</activation>
715+
<properties>
716+
<buildver>404</buildver>
717+
<spark.version>${spark404.version}</spark.version>
718+
<spark.test.version>${spark404.version}</spark.test.version>
719+
<parquet.hadoop.version>1.13.1</parquet.hadoop.version>
720+
<rapids.delta.artifactId1>rapids-4-spark-delta-40x</rapids.delta.artifactId1>
721+
<iceberg.artifact.suffix>${spark40x.iceberg.artifact.suffix}</iceberg.artifact.suffix>
722+
<iceberg.runtime.version>${iceberg.110x.version}</iceberg.runtime.version>
723+
<rapids.iceberg.artifactId>rapids-4-spark-iceberg-1-10-x</rapids.iceberg.artifactId>
724+
<rapids.iceberg.artifactId2>rapids-4-spark-iceberg-1-11-x</rapids.iceberg.artifactId2>
725+
<slf4j.version>2.0.7</slf4j.version>
726+
</properties>
727+
<modules>
728+
<module>delta-lake/delta-40x</module>
729+
<module>iceberg/iceberg-1-10-x</module>
730+
<module>iceberg/iceberg-1-11-x</module>
731+
</modules>
732+
</profile>
707733
<profile>
708734
<id>release411</id>
709735
<activation>
@@ -1035,6 +1061,7 @@
10351061
<spark401.version>4.0.1</spark401.version>
10361062
<spark402.version>4.0.2</spark402.version>
10371063
<spark403.version>4.0.3</spark403.version>
1064+
<spark404.version>4.0.4</spark404.version>
10381065
<spark411.version>4.1.1</spark411.version>
10391066
<spark412.version>4.1.2</spark412.version>
10401067
<spark413.version>4.1.3</spark413.version>
@@ -1755,14 +1782,14 @@ This will force full Scala code rebuild in downstream modules.
17551782
<requireProperty>
17561783
<property>buildver</property>
17571784
<regex>^(33[0-4]|34[0-4]|35[0-8]|341db|350db143)$</regex>
1758-
<regexMessage>Unsupported buildver value ${buildver} for a Scala 2.12 build. Supported Scala 2.12 buildver values are: 330, 331, 332, 333, 334, 340, 341, 342, 343, 344, 350, 351, 352, 353, 354, 355, 356, 357, 358, 341db, 350db143. For Scala 2.13/Spark 4.x builds, use -f scala2.13/pom.xml with a supported Scala 2.13 buildver such as 350, 400, 403, 412, 413, or 420.</regexMessage>
1785+
<regexMessage>Unsupported buildver value ${buildver} for a Scala 2.12 build. Supported Scala 2.12 buildver values are: 330, 331, 332, 333, 334, 340, 341, 342, 343, 344, 350, 351, 352, 353, 354, 355, 356, 357, 358, 341db, 350db143. For Scala 2.13/Spark 4.x builds, use -f scala2.13/pom.xml with a supported Scala 2.13 buildver such as 350, 400, 403, 404, 412, 413, or 420.</regexMessage>
17591786
</requireProperty>
17601787
--><!-- #endif scala-2.12 -->
17611788
<!-- #if scala-2.13 -->
17621789
<requireProperty>
17631790
<property>buildver</property>
1764-
<regex>^(35[0-8]|40[0-3]|41[1-3]|420|400db173)$</regex>
1765-
<regexMessage>Unsupported buildver value ${buildver} for a Scala 2.13 build. Supported Scala 2.13 buildver values are: 350, 351, 352, 353, 354, 355, 356, 357, 358, 400, 401, 402, 403, 411, 412, 413, 420, 400db173. Spark 3.3.x and 3.4.x builds use Scala 2.12; for example, build Spark 330 with the root pom.xml instead of scala2.13/pom.xml.</regexMessage>
1791+
<regex>^(35[0-8]|40[0-4]|41[1-3]|420|400db173)$</regex>
1792+
<regexMessage>Unsupported buildver value ${buildver} for a Scala 2.13 build. Supported Scala 2.13 buildver values are: 350, 351, 352, 353, 354, 355, 356, 357, 358, 400, 401, 402, 403, 404, 411, 412, 413, 420, 400db173. Spark 3.3.x and 3.4.x builds use Scala 2.12; for example, build Spark 330 with the root pom.xml instead of scala2.13/pom.xml.</regexMessage>
17661793
</requireProperty>
17671794
<!-- #endif scala-2.13 -->
17681795
<requireProperty>

0 commit comments

Comments
 (0)