Merge branch 'main' of https://github.com/nvidia/cudf-spark into cudf-spark-rename-1

Author: Sameer Raheja

datagen/src/main/spark330/scala/org/apache/spark/sql/tests/datagen/DataGenExprShims.scala

@@ -37,6 +37,7 @@
 {"spark": "356"}
 {"spark": "357"}
 {"spark": "358"}
+{"spark": "359"}
 spark-rapids-shim-json-lines ***/
 package org.apache.spark.sql.tests.datagen
 

docs/additional-functionality/advanced_configs.md

@@ -56,7 +56,7 @@ Name | Description | Default Value | Applicable at
 <a name="python.memory.gpu.allocFraction"></a>spark.rapids.python.memory.gpu.allocFraction|The fraction of total GPU memory that should be initially allocated for pooled memory for all the Python workers. It supposes to be less than (1 - $(spark.rapids.memory.gpu.allocFraction)), since the executor will share the GPU with its owning Python workers. Half of the rest will be used if not specified|None|Runtime
 <a name="python.memory.gpu.maxAllocFraction"></a>spark.rapids.python.memory.gpu.maxAllocFraction|The fraction of total GPU memory that limits the maximum size of the RMM pool for all the Python workers. It supposes to be less than (1 - $(spark.rapids.memory.gpu.maxAllocFraction)), since the executor will share the GPU with its owning Python workers. when setting to 0 it means no limit.|0.0|Runtime
 <a name="python.memory.gpu.pooling.enabled"></a>spark.rapids.python.memory.gpu.pooling.enabled|Should RMM in Python workers act as a pooling allocator for GPU memory, or should it just pass through to CUDA memory allocation directly. When not specified, It will honor the value of config 'spark.rapids.memory.gpu.pool', but now only 'DEFAULT' and 'NONE' are supported. If 'ASYNC' or 'ARENA' is specified, it will fall back to 'DEFAULT'.|None|Runtime
-<a name="shuffle.enabled"></a>spark.rapids.shuffle.enabled|Enable or disable the RAPIDS Shuffle Manager at runtime. The [RAPIDS Shuffle Manager](https://docs.nvidia.com/spark-rapids/user-guide/latest/additional-functionality/rapids-shuffle.html) must already be configured. When set to `false`, the built-in Spark shuffle will be used. |true|Runtime
+<a name="shuffle.enabled"></a>spark.rapids.shuffle.enabled|Enable or disable the RAPIDS Shuffle Manager implementation at runtime. On supported Spark versions, including Spark 4.0.0 and later, the [RAPIDS Shuffle Manager](https://docs.nvidia.com/spark-rapids/user-guide/latest/additional-functionality/rapids-shuffle.html) is configured automatically unless spark.shuffle.manager is explicitly set. On earlier Spark versions, the RAPIDS Shuffle Manager must already be configured. When set to `false`, the built-in Spark shuffle implementation will be used. |true|Runtime
 <a name="shuffle.mode"></a>spark.rapids.shuffle.mode|RAPIDS Shuffle Manager mode. "MULTITHREADED": shuffle file writes and reads are parallelized using a thread pool. "UCX": (requires UCX installation) uses accelerated transports for transferring shuffle blocks. "CACHE_ONLY": use when running a single executor, for short-circuit cached shuffle (for testing purposes).|MULTITHREADED|Startup
 <a name="shuffle.multiThreaded.maxBytesInFlight"></a>spark.rapids.shuffle.multiThreaded.maxBytesInFlight|The size limit, in bytes, that the RAPIDS shuffle manager configured in "MULTITHREADED" mode will allow to be serialized or deserialized concurrently per task. This is also the maximum amount of memory that will be used per task. This should be set larger than Spark's default maxBytesInFlight (48MB). The larger this setting is, the more compressed shuffle chunks are processed concurrently. In practice, care needs to be taken to not go over the amount of off-heap memory that Netty has available. See https://github.com/NVIDIA/cudf-spark/issues/9153.|134217728|Startup
 <a name="shuffle.multiThreaded.reader.threads"></a>spark.rapids.shuffle.multiThreaded.reader.threads|The number of threads to use for reading shuffle blocks per executor in the RAPIDS shuffle manager configured in "MULTITHREADED" mode. There are two special values: 0 = feature is disabled, falls back to Spark built-in shuffle reader; 1 = our implementation of Spark's built-in shuffle reader with extra metrics.|20|Startup

docs/download.md

@@ -46,11 +46,11 @@ The plugin is designed to work on NVIDIA Volta, Turing, Ampere, Ada Lovelace, Ho
 	Supported Spark versions:
 		Apache Spark 3.3.0, 3.3.1, 3.3.2, 3.3.3, 3.3.4
 		Apache Spark 3.4.0, 3.4.1, 3.4.2, 3.4.3, 3.4.4
-		Apache Spark 3.5.0, 3.5.1, 3.5.2, 3.5.3, 3.5.4, 3.5.5, 3.5.6, 3.5.7, 3.5.8
+		Apache Spark 3.5.0, 3.5.1, 3.5.2, 3.5.3, 3.5.4, 3.5.5, 3.5.6, 3.5.7, 3.5.8, 3.5.9
 		Apache Spark 4.0.0, 4.0.1, 4.0.2, 4.0.3, 4.0.4
 		Apache Spark 4.1.1
-		Scala 2.12: Spark 3.3.0 through 3.5.8
-		Scala 2.13: Spark 3.5.0 through 3.5.8, and Spark 4.0.0, 4.0.1, 4.0.2, 4.0.3, 4.0.4, and 4.1.1
+		Scala 2.12: Spark 3.3.0 through 3.5.9
+		Scala 2.13: Spark 3.5.0 through 3.5.9, and Spark 4.0.0, 4.0.1, 4.0.2, 4.0.3, 4.0.4, and 4.1.1
 
 	Supported Databricks runtime versions for Azure and AWS:
 		Databricks 14.3 ML LTS (GPU, Scala 2.12, Spark 3.5.0)

iceberg/README.md

@@ -12,8 +12,8 @@ and the directory that contains the corresponding support code.
 | Iceberg Version | Spark Version              | Directory         |
 |-----------------|----------------------------|-------------------|
 | 1.6.x           | Spark 3.5.0-3.5.3          | `iceberg-1-6-x`  |
-| 1.9.x           | Spark 3.5.4-3.5.8          | `iceberg-1-9-x`  |
-| 1.10.x          | Spark 3.5.4-3.5.8, 4.0.x  | `iceberg-1-10-x` |
+| 1.9.x           | Spark 3.5.4-3.5.9          | `iceberg-1-9-x`  |
+| 1.10.x          | Spark 3.5.4-3.5.9, 4.0.x  | `iceberg-1-10-x` |
 | 1.11.x          | Spark 4.0.2+, 4.1.x        | `iceberg-1-11-x` |
 
 Iceberg GPU acceleration is currently supported on Spark 3.5.x, 4.0.x, and 4.1.x.

iceberg/iceberg-1-10-x/src/main/spark354/java/com/nvidia/spark/rapids/iceberg/GpuInternalRow.java

@@ -19,6 +19,7 @@
 {"spark": "356"}
 {"spark": "357"}
 {"spark": "358"}
+{"spark": "359"}
 spark-rapids-shim-json-lines ***/
 
 package com.nvidia.spark.rapids.iceberg;

integration_tests/src/main/python/ast_test.py

@@ -17,8 +17,8 @@
 from asserts import assert_cpu_and_gpu_are_equal_collect_with_capture, assert_gpu_and_cpu_are_equal_collect
 from data_gen import *
 from marks import approximate_float, datagen_overrides, ignore_order, disable_ansi_mode
-from spark_session import with_cpu_session, is_spark_403_or_404, is_spark_412_or_later, \
-    is_spark_420_or_later
+from spark_session import with_cpu_session, is_spark_359, is_spark_403_or_404, \
+    is_spark_412_or_later, is_spark_420_or_later
 import pyspark.sql.functions as f
 
 # Each descriptor contains a list of data generators and a corresponding boolean
@@ -61,9 +61,11 @@
 
 ast_boolean_descr = [(boolean_gen, True)]
 ast_double_descr = [(double_gen, True)]
-# AST is not expressive enough to support the ACOSH Spark emulation expression in Spark 4.0.3,
-# Spark 4.0.4, and Spark 4.1.2+.
-ast_acosh_descr = [(double_gen, not (is_spark_403_or_404() or is_spark_412_or_later()))]
+# AST is not expressive enough to support the ACOSH Spark emulation expression in Spark 3.5.9,
+# Spark 4.0.3, Spark 4.0.4, and Spark 4.1.2+.
+ast_acosh_descr = [(
+    double_gen,
+    not (is_spark_359() or is_spark_403_or_404() or is_spark_412_or_later()))]
 
 _project_ast_enabled_conf = {"spark.rapids.sql.projectAstEnabled": "true"}
 

integration_tests/src/main/python/delta_lake_auto_compact_test.py

@@ -16,7 +16,7 @@
 from asserts import assert_gpu_and_cpu_writes_are_equal_collect, with_gpu_session
 from data_gen import copy_and_update, idfn
 from delta_lake_utils import *
-from marks import allow_non_gpu, delta_lake
+from marks import allow_non_gpu, delta_lake, ignore_order
 from pyspark.sql.functions import *
 from spark_session import is_spark_353_or_later, is_databricks_runtime, \
     is_databricks104_or_later, is_databricks173_or_later, supports_delta_lake_deletion_vectors
@@ -111,6 +111,8 @@ def test_auto_compact_dbr173_inline_uses_gpu(spark_tmp_path):
     with_cpu_session(lambda spark: assert_optimized(spark, data_path), {})
 
 
+# Auto-configured RapidsShuffleManager may change row ordering.
+@ignore_order(local=True)
 @delta_lake
 @allow_non_gpu(*delta_meta_allow)
 @pytest.mark.skipif(not is_databricks_runtime() and not is_spark_353_or_later(),

integration_tests/src/main/python/spark_session.py

@@ -243,6 +243,9 @@ def is_spark_351_or_later():
 def is_spark_356_or_later():
     return spark_version() >= "3.5.6"
 
+def is_spark_359():
+    return spark_version() == "3.5.9"
+
 def is_spark_35x():
     return "3.5.0" <= spark_version() < "3.6.0"
 

integration_tests/src/test/spark330/scala/org/apache/spark/sql/rapids/shims/TrampolineUtilShim.scala

@@ -38,6 +38,7 @@
 {"spark": "356"}
 {"spark": "357"}
 {"spark": "358"}
+{"spark": "359"}
 spark-rapids-shim-json-lines ***/
 package org.apache.spark.sql.rapids.shims
 

pom.xml

@@ -557,6 +557,32 @@
                 <module>iceberg/iceberg-1-10-x</module>
             </modules>
         </profile>
+        <profile>
+            <id>release359</id>
+            <activation>
+                <property>
+                    <name>buildver</name>
+                    <value>359</value>
+                </property>
+            </activation>
+            <properties>
+                <buildver>359</buildver>
+                <spark.version>${spark359.version}</spark.version>
+                <spark.test.version>${spark359.version}</spark.test.version>
+                <parquet.hadoop.version>1.13.1</parquet.hadoop.version>
+                <rapids.delta.artifactId1>rapids-4-spark-delta-33x</rapids.delta.artifactId1>
+                <iceberg.artifact.suffix>${spark35x.iceberg.artifact.suffix}</iceberg.artifact.suffix>
+                <iceberg.version>${iceberg.19x.version}</iceberg.version>
+                <rapids.iceberg.artifactId>rapids-4-spark-iceberg-1-9-x</rapids.iceberg.artifactId>
+                <rapids.iceberg.artifactId2>rapids-4-spark-iceberg-1-10-x</rapids.iceberg.artifactId2>
+                <slf4j.version>2.0.7</slf4j.version>
+            </properties>
+            <modules>
+                <module>delta-lake/delta-33x</module>
+                <module>iceberg/iceberg-1-9-x</module>
+                <module>iceberg/iceberg-1-10-x</module>
+            </modules>
+        </profile>
 	<!-- #if scala-2.13 --><!--
         <profile>
             <id>release400</id>
@@ -1032,6 +1058,7 @@
         <spark356.version>3.5.6</spark356.version>
         <spark357.version>3.5.7</spark357.version>
         <spark358.version>3.5.8</spark358.version>
+        <spark359.version>3.5.9</spark359.version>
         <spark400.version>4.0.0</spark400.version>
         <spark400db173.version>4.0.0-databricks-173</spark400db173.version>
         <spark401.version>4.0.1</spark401.version>
@@ -1757,15 +1784,15 @@ This will force full Scala code rebuild in downstream modules.
                         <!-- #if scala-2.12 -->
                         <requireProperty>
                           <property>buildver</property>
-                          <regex>^(33[0-4]|34[0-4]|35[0-8]|350db143)$</regex>
-                          <regexMessage>Unsupported buildver value ${buildver} for a Scala 2.12 build. Supported Scala 2.12 buildver values are: 330, 331, 332, 333, 334, 340, 341, 342, 343, 344, 350, 351, 352, 353, 354, 355, 356, 357, 358, 350db143. For Scala 2.13/Spark 4.x builds, use -f scala2.13/pom.xml with a supported Scala 2.13 buildver such as 350, 400, 403, 404, 412, 413, or 420.</regexMessage>
+                          <regex>^(33[0-4]|34[0-4]|35[0-9]|350db143)$</regex>
+                          <regexMessage>Unsupported buildver value ${buildver} for a Scala 2.12 build. Supported Scala 2.12 buildver values are: 330, 331, 332, 333, 334, 340, 341, 342, 343, 344, 350, 351, 352, 353, 354, 355, 356, 357, 358, 359, 350db143. For Scala 2.13/Spark 4.x builds, use -f scala2.13/pom.xml with a supported Scala 2.13 buildver such as 350, 400, 403, 404, 412, 413, or 420.</regexMessage>
                         </requireProperty>
                         <!-- #endif scala-2.12 -->
                         <!-- #if scala-2.13 --><!--
                         <requireProperty>
                             <property>buildver</property>
-                            <regex>^(35[0-8]|40[0-4]|41[1-3]|420|400db173)$</regex>
-                            <regexMessage>Unsupported buildver value ${buildver} for a Scala 2.13 build. Supported Scala 2.13 buildver values are: 350, 351, 352, 353, 354, 355, 356, 357, 358, 400, 401, 402, 403, 404, 411, 412, 413, 420, 400db173. Spark 3.3.x and 3.4.x builds use Scala 2.12; for example, build Spark 330 with the root pom.xml instead of scala2.13/pom.xml.</regexMessage>
+                            <regex>^(35[0-9]|40[0-4]|41[1-3]|420|400db173)$</regex>
+                            <regexMessage>Unsupported buildver value ${buildver} for a Scala 2.13 build. Supported Scala 2.13 buildver values are: 350, 351, 352, 353, 354, 355, 356, 357, 358, 359, 400, 401, 402, 403, 404, 411, 412, 413, 420, 400db173. Spark 3.3.x and 3.4.x builds use Scala 2.12; for example, build Spark 330 with the root pom.xml instead of scala2.13/pom.xml.</regexMessage>
                         </requireProperty>
                         --><!-- #endif scala-2.13 -->
                         <requireProperty>