Skip to content

Commit 843b8a0

Browse files
firestarmanfirestarman
authored
Add Spark 4.1.3 shim support[databricks] (NVIDIA#15310)
Fixes NVIDIA#15271. ### Description - Add Spark 4.1.3 build metadata, release profile, and Scala 2.13 buildver validation so the public plugin can build against the new Spark shim. - Extend compatible Spark shim annotations to include 4.1.3 so existing shared Spark 4.1.x shim implementations are compiled for the new buildver. - Add Spark 4.1.3-specific shim service provider and shims suite because the provider/test version checks need to report 4.1.3 explicitly. - Update generated operator metadata for buildver 413 so tool output matches the newly supported Spark version. - Validated with `mvn -f scala2.13/pom.xml -Dbuildver=413 -Dcuda.version=cuda13 clean package`: Scala tests passed for `sql-plugin` (64), `udf-compiler` (165), `integration_tests` (36), and `tests` (1821 passed, 104 canceled, 31 ignored). - Local NDS performance results show no overall performance regression observed for Spark 4.1.3. | Item | Value | | ------------- | ---------------------------------------------------------- | | Dataset | /bigdata/tpcds_data/parquet_100f | | Format | Parquet | | RAPIDS jar | rapids-4-spark_2.13-26.08.0-SNAPSHOT-cuda13-412-413.jar | | Spark 4.1.2 | spark-4.1.2-bin-hadoop3 | | Spark 4.1.3 | spark-4.1.3-bin-hadoop3 | | Runs | 3 per shim | | Compared runs | Warm runs only: run 2 and run 3 | | Query status | 618 query JSON records checked, 0 non-Completed | | Result | No overall performance regression observed for Spark 4.1.3 | | Metric | Spark 4.1.2 | Spark 4.1.3 | 4.1.3 vs 4.1.2 | | -------------------------- | ----------- | ----------- | -------------- | | Power run 2 | 577.000s | 621.000s | +7.63% | | Power run 3 | 589.000s | 541.000s | -8.15% | | Power run avg | 583.000s | 581.000s | -0.34% | | Sum of per-query avg times | 581.716s | 579.863s | -0.32% | ### Checklists Documentation - [x] Updated for new or modified user-facing features or behaviors - [ ] No user-facing change Testing - [ ] Added or modified tests to cover new code paths - [x] Covered by existing tests (Please provide the names of the existing tests in the PR description.) - [ ] Not required Performance - [x] Tests ran and results are added in the PR description - [ ] Issue filed with a link in the PR description - [ ] Not required Signed-off-by: Firestarman <firestarmanllc@gmail.com>
1 parent 1f3df51 commit 843b8a0

207 files changed

Lines changed: 1728 additions & 6 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎datagen/src/main/spark400/scala/org/apache/spark/sql/tests/datagen/DataGenExprShims.scala‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,7 @@
2222
{"spark": "403"}
2323
{"spark": "411"}
2424
{"spark": "412"}
25+
{"spark": "413"}
2526
{"spark": "420"}
2627
spark-rapids-shim-json-lines ***/
2728
package org.apache.spark.sql.tests.datagen

‎integration_tests/src/test/spark400/scala/org/apache/spark/sql/rapids/shims/TrampolineUtilShim.scala‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,7 @@
2222
{"spark": "403"}
2323
{"spark": "411"}
2424
{"spark": "412"}
25+
{"spark": "413"}
2526
{"spark": "420"}
2627
spark-rapids-shim-json-lines ***/
2728

‎pom.xml‎

Lines changed: 42 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -781,6 +781,44 @@
781781
</pluginManagement>
782782
</build>
783783
</profile>
784+
<profile>
785+
<id>release413</id>
786+
<activation>
787+
<property>
788+
<name>buildver</name>
789+
<value>413</value>
790+
</property>
791+
</activation>
792+
<properties>
793+
<buildver>413</buildver>
794+
<spark.version>${spark413.version}</spark.version>
795+
<spark.test.version>${spark413.version}</spark.test.version>
796+
<parquet.hadoop.version>1.13.1</parquet.hadoop.version>
797+
<rapids.delta.artifactId1>rapids-4-spark-delta-41x</rapids.delta.artifactId1>
798+
<iceberg.artifact.suffix>${spark40x.iceberg.artifact.suffix}</iceberg.artifact.suffix>
799+
<iceberg.runtime.version>${iceberg.110x.version}</iceberg.runtime.version>
800+
<slf4j.version>2.0.7</slf4j.version>
801+
<scala.javac.args>-Xlint:all,-serial,-path,-try,-processing,-options|-Werror</scala.javac.args>
802+
</properties>
803+
<modules>
804+
<module>delta-lake/delta-41x</module>
805+
<module>iceberg/iceberg-stub</module>
806+
</modules>
807+
<build>
808+
<pluginManagement>
809+
<plugins>
810+
<plugin>
811+
<groupId>net.alchim31.maven</groupId>
812+
<artifactId>scala-maven-plugin</artifactId>
813+
<configuration>
814+
<release combine.self="override"/>
815+
<target>${java.major.version}</target>
816+
</configuration>
817+
</plugin>
818+
</plugins>
819+
</pluginManagement>
820+
</build>
821+
</profile>
784822
<profile>
785823
<id>release420</id>
786824
<activation>
@@ -999,6 +1037,7 @@
9991037
<spark403.version>4.0.3</spark403.version>
10001038
<spark411.version>4.1.1</spark411.version>
10011039
<spark412.version>4.1.2</spark412.version>
1040+
<spark413.version>4.1.3</spark413.version>
10021041
<spark420.version>4.2.0</spark420.version>
10031042
<mockito.version>3.12.4</mockito.version>
10041043
<!-- 4.7.1 required: later versions auto-derive -release from -target,
@@ -1716,14 +1755,14 @@ This will force full Scala code rebuild in downstream modules.
17161755
<requireProperty>
17171756
<property>buildver</property>
17181757
<regex>^(33[0-4]|34[0-4]|35[0-8]|341db|350db143)$</regex>
1719-
<regexMessage>Unsupported buildver value ${buildver} for a Scala 2.12 build. Supported Scala 2.12 buildver values are: 330, 331, 332, 333, 334, 340, 341, 342, 343, 344, 350, 351, 352, 353, 354, 355, 356, 357, 358, 341db, 350db143. For Scala 2.13/Spark 4.x builds, use -f scala2.13/pom.xml with a supported Scala 2.13 buildver such as 350, 400, 403, 412, or 420.</regexMessage>
1758+
<regexMessage>Unsupported buildver value ${buildver} for a Scala 2.12 build. Supported Scala 2.12 buildver values are: 330, 331, 332, 333, 334, 340, 341, 342, 343, 344, 350, 351, 352, 353, 354, 355, 356, 357, 358, 341db, 350db143. For Scala 2.13/Spark 4.x builds, use -f scala2.13/pom.xml with a supported Scala 2.13 buildver such as 350, 400, 403, 412, 413, or 420.</regexMessage>
17201759
</requireProperty>
17211760
<!-- #endif scala-2.12 -->
17221761
<!-- #if scala-2.13 --><!--
17231762
<requireProperty>
17241763
<property>buildver</property>
1725-
<regex>^(35[0-8]|40[0-3]|41[1-2]|420|400db173)$</regex>
1726-
<regexMessage>Unsupported buildver value ${buildver} for a Scala 2.13 build. Supported Scala 2.13 buildver values are: 350, 351, 352, 353, 354, 355, 356, 357, 358, 400, 401, 402, 403, 411, 412, 420, 400db173. Spark 3.3.x and 3.4.x builds use Scala 2.12; for example, build Spark 330 with the root pom.xml instead of scala2.13/pom.xml.</regexMessage>
1764+
<regex>^(35[0-8]|40[0-3]|41[1-3]|420|400db173)$</regex>
1765+
<regexMessage>Unsupported buildver value ${buildver} for a Scala 2.13 build. Supported Scala 2.13 buildver values are: 350, 351, 352, 353, 354, 355, 356, 357, 358, 400, 401, 402, 403, 411, 412, 413, 420, 400db173. Spark 3.3.x and 3.4.x builds use Scala 2.12; for example, build Spark 330 with the root pom.xml instead of scala2.13/pom.xml.</regexMessage>
17271766
</requireProperty>
17281767
--><!-- #endif scala-2.13 -->
17291768
<requireProperty>

‎scala2.13/pom.xml‎

Lines changed: 42 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -781,6 +781,44 @@
781781
</pluginManagement>
782782
</build>
783783
</profile>
784+
<profile>
785+
<id>release413</id>
786+
<activation>
787+
<property>
788+
<name>buildver</name>
789+
<value>413</value>
790+
</property>
791+
</activation>
792+
<properties>
793+
<buildver>413</buildver>
794+
<spark.version>${spark413.version}</spark.version>
795+
<spark.test.version>${spark413.version}</spark.test.version>
796+
<parquet.hadoop.version>1.13.1</parquet.hadoop.version>
797+
<rapids.delta.artifactId1>rapids-4-spark-delta-41x</rapids.delta.artifactId1>
798+
<iceberg.artifact.suffix>${spark40x.iceberg.artifact.suffix}</iceberg.artifact.suffix>
799+
<iceberg.runtime.version>${iceberg.110x.version}</iceberg.runtime.version>
800+
<slf4j.version>2.0.7</slf4j.version>
801+
<scala.javac.args>-Xlint:all,-serial,-path,-try,-processing,-options|-Werror</scala.javac.args>
802+
</properties>
803+
<modules>
804+
<module>delta-lake/delta-41x</module>
805+
<module>iceberg/iceberg-stub</module>
806+
</modules>
807+
<build>
808+
<pluginManagement>
809+
<plugins>
810+
<plugin>
811+
<groupId>net.alchim31.maven</groupId>
812+
<artifactId>scala-maven-plugin</artifactId>
813+
<configuration>
814+
<release combine.self="override"/>
815+
<target>${java.major.version}</target>
816+
</configuration>
817+
</plugin>
818+
</plugins>
819+
</pluginManagement>
820+
</build>
821+
</profile>
784822
<profile>
785823
<id>release420</id>
786824
<activation>
@@ -999,6 +1037,7 @@
9991037
<spark403.version>4.0.3</spark403.version>
10001038
<spark411.version>4.1.1</spark411.version>
10011039
<spark412.version>4.1.2</spark412.version>
1040+
<spark413.version>4.1.3</spark413.version>
10021041
<spark420.version>4.2.0</spark420.version>
10031042
<mockito.version>3.12.4</mockito.version>
10041043
<!-- 4.7.1 required: later versions auto-derive -release from -target,
@@ -1716,14 +1755,14 @@ This will force full Scala code rebuild in downstream modules.
17161755
<requireProperty>
17171756
<property>buildver</property>
17181757
<regex>^(33[0-4]|34[0-4]|35[0-8]|341db|350db143)$</regex>
1719-
<regexMessage>Unsupported buildver value ${buildver} for a Scala 2.12 build. Supported Scala 2.12 buildver values are: 330, 331, 332, 333, 334, 340, 341, 342, 343, 344, 350, 351, 352, 353, 354, 355, 356, 357, 358, 341db, 350db143. For Scala 2.13/Spark 4.x builds, use -f scala2.13/pom.xml with a supported Scala 2.13 buildver such as 350, 400, 403, 412, or 420.</regexMessage>
1758+
<regexMessage>Unsupported buildver value ${buildver} for a Scala 2.12 build. Supported Scala 2.12 buildver values are: 330, 331, 332, 333, 334, 340, 341, 342, 343, 344, 350, 351, 352, 353, 354, 355, 356, 357, 358, 341db, 350db143. For Scala 2.13/Spark 4.x builds, use -f scala2.13/pom.xml with a supported Scala 2.13 buildver such as 350, 400, 403, 412, 413, or 420.</regexMessage>
17201759
</requireProperty>
17211760
--><!-- #endif scala-2.12 -->
17221761
<!-- #if scala-2.13 -->
17231762
<requireProperty>
17241763
<property>buildver</property>
1725-
<regex>^(35[0-8]|40[0-3]|41[1-2]|420|400db173)$</regex>
1726-
<regexMessage>Unsupported buildver value ${buildver} for a Scala 2.13 build. Supported Scala 2.13 buildver values are: 350, 351, 352, 353, 354, 355, 356, 357, 358, 400, 401, 402, 403, 411, 412, 420, 400db173. Spark 3.3.x and 3.4.x builds use Scala 2.12; for example, build Spark 330 with the root pom.xml instead of scala2.13/pom.xml.</regexMessage>
1764+
<regex>^(35[0-8]|40[0-3]|41[1-3]|420|400db173)$</regex>
1765+
<regexMessage>Unsupported buildver value ${buildver} for a Scala 2.13 build. Supported Scala 2.13 buildver values are: 350, 351, 352, 353, 354, 355, 356, 357, 358, 400, 401, 402, 403, 411, 412, 413, 420, 400db173. Spark 3.3.x and 3.4.x builds use Scala 2.12; for example, build Spark 330 with the root pom.xml instead of scala2.13/pom.xml.</regexMessage>
17271766
</requireProperty>
17281767
<!-- #endif scala-2.13 -->
17291768
<requireProperty>

‎scala2.13/sql-plugin-api/pom.xml‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -108,5 +108,28 @@
108108
</pluginManagement>
109109
</build>
110110
</profile>
111+
<profile>
112+
<id>release413</id>
113+
<activation>
114+
<property>
115+
<name>buildver</name>
116+
<value>413</value>
117+
</property>
118+
</activation>
119+
<build>
120+
<pluginManagement>
121+
<plugins>
122+
<plugin>
123+
<groupId>net.alchim31.maven</groupId>
124+
<artifactId>scala-maven-plugin</artifactId>
125+
<configuration>
126+
<release conf.combine="override">${java.major.version}</release>
127+
<addJavacArgs conf.combine="override">-Xlint:all,-serial,-path,-try,-processing</addJavacArgs>
128+
</configuration>
129+
</plugin>
130+
</plugins>
131+
</pluginManagement>
132+
</build>
133+
</profile>
111134
</profiles>
112135
</project>

‎scala2.13/tests/pom.xml‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -257,6 +257,29 @@
257257
</dependency>
258258
</dependencies>
259259
</profile>
260+
<profile>
261+
<id>release413</id>
262+
<activation>
263+
<property>
264+
<name>buildver</name>
265+
<value>413</value>
266+
</property>
267+
</activation>
268+
<dependencies>
269+
<dependency>
270+
<groupId>io.delta</groupId>
271+
<artifactId>${delta41x.spark.artifactId}</artifactId>
272+
<version>${delta41x.version}</version>
273+
<scope>test</scope>
274+
<exclusions>
275+
<exclusion>
276+
<groupId>io.unitycatalog</groupId>
277+
<artifactId>unitycatalog-client</artifactId>
278+
</exclusion>
279+
</exclusions>
280+
</dependency>
281+
</dependencies>
282+
</profile>
260283
<!-- #endif scala-2.13 -->
261284

262285
</profiles>

‎sql-plugin-api/pom.xml‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -108,5 +108,28 @@
108108
</pluginManagement>
109109
</build>
110110
</profile>
111+
<profile>
112+
<id>release413</id>
113+
<activation>
114+
<property>
115+
<name>buildver</name>
116+
<value>413</value>
117+
</property>
118+
</activation>
119+
<build>
120+
<pluginManagement>
121+
<plugins>
122+
<plugin>
123+
<groupId>net.alchim31.maven</groupId>
124+
<artifactId>scala-maven-plugin</artifactId>
125+
<configuration>
126+
<release conf.combine="override">${java.major.version}</release>
127+
<addJavacArgs conf.combine="override">-Xlint:all,-serial,-path,-try,-processing</addJavacArgs>
128+
</configuration>
129+
</plugin>
130+
</plugins>
131+
</pluginManagement>
132+
</build>
133+
</profile>
111134
</profiles>
112135
</project>

‎sql-plugin/src/main/spark330/scala/com/nvidia/spark/rapids/GpuInSubqueryExec.scala‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -40,6 +40,7 @@
4040
{"spark": "403"}
4141
{"spark": "411"}
4242
{"spark": "412"}
43+
{"spark": "413"}
4344
{"spark": "420"}
4445
spark-rapids-shim-json-lines ***/
4546
package com.nvidia.spark.rapids

‎sql-plugin/src/main/spark330/scala/com/nvidia/spark/rapids/shims/AQEUtils.scala‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -40,6 +40,7 @@
4040
{"spark": "403"}
4141
{"spark": "411"}
4242
{"spark": "412"}
43+
{"spark": "413"}
4344
{"spark": "420"}
4445
spark-rapids-shim-json-lines ***/
4546
package com.nvidia.spark.rapids.shims

‎sql-plugin/src/main/spark330/scala/com/nvidia/spark/rapids/shims/AggregationTagging.scala‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -40,6 +40,7 @@
4040
{"spark": "403"}
4141
{"spark": "411"}
4242
{"spark": "412"}
43+
{"spark": "413"}
4344
{"spark": "420"}
4445
spark-rapids-shim-json-lines ***/
4546
package com.nvidia.spark.rapids.shims

0 commit comments

Comments
 (0)