Skip to content

Commit 0259aa9

Browse files
igorpeshanskyigorpeshansky
committed
Merge branch 'main' into ci-allowlist-normalize
Signed-off-by: Igor Peshansky <ipeshansky@nvidia.com>
2 parents 96514dd + 5477e23 commit 0259aa9

12 files changed

Lines changed: 994 additions & 84 deletions

File tree

‎.github/workflows/blossom-ci.yml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -47,6 +47,7 @@ jobs:
4747
github.actor == 'gerashegalov' ||
4848
github.actor == 'HaoYang670' ||
4949
github.actor == 'hyperbolic2346' ||
50+
github.actor == 'igorpeshansky' ||
5051
github.actor == 'jihoonson' ||
5152
github.actor == 'knoguchi22' ||
5253
github.actor == 'kuhushukla' ||

‎docs/archive.md‎

Lines changed: 103 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,109 @@ nav_order: 15
55
---
66
Below are archived releases for RAPIDS Accelerator for Apache Spark.
77

8+
## Release v26.06.0
9+
### Hardware Requirements:
10+
11+
The plugin is designed to work on NVIDIA Volta, Turing, Ampere, Ada Lovelace, Hopper and Blackwell generation datacenter GPUs. The plugin jar is tested on the following GPUs:
12+
13+
GPU Models: NVIDIA V100, T4, A10, A100, L4, H100 and B100 GPUs
14+
15+
### Software Requirements:
16+
17+
OS: Spark RAPIDS is compatible with any Linux distribution with glibc >= 2.28 (Please check ldd --version output). glibc 2.28 was released August 1, 2018.
18+
Tested on Ubuntu 22.04, Ubuntu 24.04, Rocky Linux 8 and Rocky Linux 9
19+
20+
NVIDIA Driver*: R525+
21+
22+
Runtime:
23+
Scala 2.12, 2.13
24+
Python, Java Virtual Machine (JVM) compatible with your spark-version.
25+
26+
* Check the Spark documentation for Python and Java version compatibility with your specific
27+
Spark version. For instance, visit `https://spark.apache.org/docs/3.4.1` for Spark 3.4.1.
28+
29+
Supported Spark versions:
30+
Apache Spark 3.3.0, 3.3.1, 3.3.2, 3.3.3, 3.3.4
31+
Apache Spark 3.4.0, 3.4.1, 3.4.2, 3.4.3, 3.4.4
32+
Apache Spark 3.5.0, 3.5.1, 3.5.2, 3.5.3, 3.5.4, 3.5.5, 3.5.6, 3.5.7, 3.5.8
33+
Apache Spark 4.0.0, 4.0.1, 4.0.2, 4.0.3
34+
Apache Spark 4.1.1
35+
Scala 2.12: Spark 3.3.0 through 3.5.8
36+
Scala 2.13: Spark 3.5.0 through 3.5.8, and Spark 4.0.0, 4.0.1, 4.0.2, 4.0.3, and 4.1.1
37+
38+
Supported Databricks runtime versions for Azure and AWS:
39+
Databricks 13.3 ML LTS (GPU, Scala 2.12, Spark 3.4.1)
40+
Databricks 14.3 ML LTS (GPU, Scala 2.12, Spark 3.5.0)
41+
Databricks 17.3 ML LTS (GPU, Scala 2.13, Spark 4.0.0)
42+
43+
Supported Dataproc versions (Debian/Ubuntu/Rocky):
44+
GCP Dataproc 2.1
45+
GCP Dataproc 2.2
46+
GCP Dataproc 2.3
47+
48+
Supported Dataproc Serverless versions:
49+
Spark runtime 1.2 LTS
50+
Spark runtime 2.2 LTS
51+
Spark runtime 2.3 LTS
52+
Spark runtime 3.0
53+
54+
*Some hardware may have a minimum driver version greater than R470. Check the GPU spec sheet
55+
for your hardware's minimum driver version.
56+
57+
*For EMR support, please refer to the
58+
[Distributions](https://docs.nvidia.com/spark-rapids/user-guide/latest/faq.html#which-distributions-are-supported) section of the FAQ.
59+
60+
### RAPIDS Accelerator's Support Policy for Apache Spark
61+
The RAPIDS Accelerator maintains support for Apache Spark versions available for download from [Apache Spark](https://spark.apache.org/downloads.html)
62+
63+
### Download RAPIDS Accelerator for Apache Spark v26.06.0
64+
65+
#### CUDA 12
66+
67+
| Processor | Scala Version | Download Jar | Download Signature | Download From Maven |
68+
|-----------|---------------|--------------|--------------------|---------------------|
69+
| x86_64 | Scala 2.12 | [RAPIDS Accelerator v26.06.0](https://repo1.maven.org/maven2/com/nvidia/rapids-4-spark_2.12/26.06.0/rapids-4-spark_2.12-26.06.0.jar) | [Signature](https://repo1.maven.org/maven2/com/nvidia/rapids-4-spark_2.12/26.06.0/rapids-4-spark_2.12-26.06.0.jar.asc) | <pre>&lt;dependency&gt;<br/> &lt;groupId&gt;com.nvidia&lt;/groupId&gt;<br/> &lt;artifactId&gt;rapids-4-spark_2.12&lt;/artifactId&gt;<br/> &lt;version&gt;26.06.0&lt;/version&gt;<br/>&lt;/dependency&gt;</pre> |
70+
| x86_64 | Scala 2.13 | [RAPIDS Accelerator v26.06.0](https://repo1.maven.org/maven2/com/nvidia/rapids-4-spark_2.13/26.06.0/rapids-4-spark_2.13-26.06.0.jar) | [Signature](https://repo1.maven.org/maven2/com/nvidia/rapids-4-spark_2.13/26.06.0/rapids-4-spark_2.13-26.06.0.jar.asc) | <pre>&lt;dependency&gt;<br/> &lt;groupId&gt;com.nvidia&lt;/groupId&gt;<br/> &lt;artifactId&gt;rapids-4-spark_2.13&lt;/artifactId&gt;<br/> &lt;version&gt;26.06.0&lt;/version&gt;<br/>&lt;/dependency&gt;</pre> |
71+
| arm64 | Scala 2.12 | [RAPIDS Accelerator v26.06.0](https://repo1.maven.org/maven2/com/nvidia/rapids-4-spark_2.12/26.06.0/rapids-4-spark_2.12-26.06.0-cuda12-arm64.jar) | [Signature](https://repo1.maven.org/maven2/com/nvidia/rapids-4-spark_2.12/26.06.0/rapids-4-spark_2.12-26.06.0-cuda12-arm64.jar.asc) | <pre>&lt;dependency&gt;<br/> &lt;groupId&gt;com.nvidia&lt;/groupId&gt;<br/> &lt;artifactId&gt;rapids-4-spark_2.12&lt;/artifactId&gt;<br/> &lt;version&gt;26.06.0&lt;/version&gt;<br/> &lt;classifier&gt;cuda12-arm64&lt;/classifier&gt;<br/>&lt;/dependency&gt;</pre> |
72+
| arm64 | Scala 2.13 | [RAPIDS Accelerator v26.06.0](https://repo1.maven.org/maven2/com/nvidia/rapids-4-spark_2.13/26.06.0/rapids-4-spark_2.13-26.06.0-cuda12-arm64.jar) | [Signature](https://repo1.maven.org/maven2/com/nvidia/rapids-4-spark_2.13/26.06.0/rapids-4-spark_2.13-26.06.0-cuda12-arm64.jar.asc) | <pre>&lt;dependency&gt;<br/> &lt;groupId&gt;com.nvidia&lt;/groupId&gt;<br/> &lt;artifactId&gt;rapids-4-spark_2.13&lt;/artifactId&gt;<br/> &lt;version&gt;26.06.0&lt;/version&gt;<br/> &lt;classifier&gt;cuda12-arm64&lt;/classifier&gt;<br/>&lt;/dependency&gt;</pre> |
73+
74+
75+
#### CUDA 13
76+
77+
| Processor | Scala Version | Download Jar | Download Signature | Download From Maven |
78+
|-----------|---------------|--------------|--------------------|---------------------|
79+
| x86_64 | Scala 2.12 | [RAPIDS Accelerator v26.06.0](https://repo1.maven.org/maven2/com/nvidia/rapids-4-spark_2.12/26.06.0/rapids-4-spark_2.12-26.06.0-cuda13.jar) | [Signature](https://repo1.maven.org/maven2/com/nvidia/rapids-4-spark_2.12/26.06.0/rapids-4-spark_2.12-26.06.0-cuda13.jar.asc) | <pre>&lt;dependency&gt;<br/> &lt;groupId&gt;com.nvidia&lt;/groupId&gt;<br/> &lt;artifactId&gt;rapids-4-spark_2.12&lt;/artifactId&gt;<br/> &lt;version&gt;26.06.0&lt;/version&gt;<br/> &lt;classifier&gt;cuda13&lt;/classifier&gt;<br/>&lt;/dependency&gt;</pre> |
80+
| x86_64 | Scala 2.13 | [RAPIDS Accelerator v26.06.0](https://repo1.maven.org/maven2/com/nvidia/rapids-4-spark_2.13/26.06.0/rapids-4-spark_2.13-26.06.0-cuda13.jar) | [Signature](https://repo1.maven.org/maven2/com/nvidia/rapids-4-spark_2.13/26.06.0/rapids-4-spark_2.13-26.06.0-cuda13.jar.asc) | <pre>&lt;dependency&gt;<br/> &lt;groupId&gt;com.nvidia&lt;/groupId&gt;<br/> &lt;artifactId&gt;rapids-4-spark_2.13&lt;/artifactId&gt;<br/> &lt;version&gt;26.06.0&lt;/version&gt;<br/> &lt;classifier&gt;cuda13&lt;/classifier&gt;<br/>&lt;/dependency&gt;</pre> |
81+
| arm64 | Scala 2.12 | [RAPIDS Accelerator v26.06.0](https://repo1.maven.org/maven2/com/nvidia/rapids-4-spark_2.12/26.06.0/rapids-4-spark_2.12-26.06.0-cuda13-arm64.jar) | [Signature](https://repo1.maven.org/maven2/com/nvidia/rapids-4-spark_2.12/26.06.0/rapids-4-spark_2.12-26.06.0-cuda13-arm64.jar.asc) | <pre>&lt;dependency&gt;<br/> &lt;groupId&gt;com.nvidia&lt;/groupId&gt;<br/> &lt;artifactId&gt;rapids-4-spark_2.12&lt;/artifactId&gt;<br/> &lt;version&gt;26.06.0&lt;/version&gt;<br/> &lt;classifier&gt;cuda13-arm64&lt;/classifier&gt;<br/>&lt;/dependency&gt;</pre> |
82+
| arm64 | Scala 2.13 | [RAPIDS Accelerator v26.06.0](https://repo1.maven.org/maven2/com/nvidia/rapids-4-spark_2.13/26.06.0/rapids-4-spark_2.13-26.06.0-cuda13-arm64.jar) | [Signature](https://repo1.maven.org/maven2/com/nvidia/rapids-4-spark_2.13/26.06.0/rapids-4-spark_2.13-26.06.0-cuda13-arm64.jar.asc) | <pre>&lt;dependency&gt;<br/> &lt;groupId&gt;com.nvidia&lt;/groupId&gt;<br/> &lt;artifactId&gt;rapids-4-spark_2.13&lt;/artifactId&gt;<br/> &lt;version&gt;26.06.0&lt;/version&gt;<br/> &lt;classifier&gt;cuda13-arm64&lt;/classifier&gt;<br/>&lt;/dependency&gt;</pre> |
83+
84+
85+
The above packages are built against CUDA 12.9 or CUDA 13.1. They are tested on V100, T4, A10, A100, L4, H100 and GB100 GPUs.
86+
87+
### Verify signature
88+
* Download the [PUB_KEY](https://keys.openpgp.org/search?q=sw-spark@nvidia.com).
89+
* Import the public key: `gpg --import PUB_KEY`
90+
* Verify the signature for Scala 2.12 jar:
91+
`gpg --verify rapids-4-spark_2.12-26.06.0.jar.asc rapids-4-spark_2.12-26.06.0.jar`
92+
* Verify the signature for Scala 2.13 jar:
93+
`gpg --verify rapids-4-spark_2.13-26.06.0.jar.asc rapids-4-spark_2.13-26.06.0.jar`
94+
95+
The output of signature verify:
96+
97+
gpg: Good signature from "NVIDIA Spark (For the signature of spark-rapids release jars) <sw-spark@nvidia.com>"
98+
99+
### Release Notes
100+
v26.06.0 includes the following updates:
101+
* Databricks 17.3 ML LTS Delta Lake support now includes native deletion vector reads, Delta writes, DELETE, UPDATE, MERGE, OPTIMIZE, and auto compaction on GPU ([#14787](https://github.com/NVIDIA/spark-rapids/pull/14787), [#14716](https://github.com/NVIDIA/spark-rapids/pull/14716), [#14810](https://github.com/NVIDIA/spark-rapids/pull/14810), [#14820](https://github.com/NVIDIA/spark-rapids/pull/14820), [#14847](https://github.com/NVIDIA/spark-rapids/pull/14847))
102+
* Iceberg support adds nested and binary GPU writes, read-path optimizations, per-table scan-option overrides, and fixes for newly-added nested MAP/LIST fields ([#14611](https://github.com/NVIDIA/spark-rapids/pull/14611), [#14674](https://github.com/NVIDIA/spark-rapids/pull/14674), [#14754](https://github.com/NVIDIA/spark-rapids/pull/14754), [#14880](https://github.com/NVIDIA/spark-rapids/pull/14880))
103+
* Added GPU support for array aggregate SUM, PRODUCT, MAX, MIN, ALL, and ANY, plus additional string expression support for `replace(col, targetExpr, replExpr)` and GBK `StringDecode` ([#14652](https://github.com/NVIDIA/spark-rapids/pull/14652), [#14623](https://github.com/NVIDIA/spark-rapids/pull/14623), [#14545](https://github.com/NVIDIA/spark-rapids/pull/14545))
104+
* Improved expression performance for timestamp parsing, complex type casts, null handling, `hypot`, `format_number`, regex extract, and substring operations ([#14706](https://github.com/NVIDIA/spark-rapids/pull/14706), [#14842](https://github.com/NVIDIA/spark-rapids/pull/14842), [#14817](https://github.com/NVIDIA/spark-rapids/pull/14817), [#14818](https://github.com/NVIDIA/spark-rapids/pull/14818), [#14830](https://github.com/NVIDIA/spark-rapids/pull/14830), [#14586](https://github.com/NVIDIA/spark-rapids/pull/14586), [#14647](https://github.com/NVIDIA/spark-rapids/pull/14647), [#14819](https://github.com/NVIDIA/spark-rapids/pull/14819))
105+
* Improved retry and resource handling for GPU project execution, row-to-column transitions, asynchronous cloud output writes, and host column extraction ([#14724](https://github.com/NVIDIA/spark-rapids/pull/14724), [#14865](https://github.com/NVIDIA/spark-rapids/pull/14865), [#14759](https://github.com/NVIDIA/spark-rapids/pull/14759))
106+
* Fixed several query correctness and compatibility issues, including join conditions with casts, non-deterministic expression preservation, JSON/CSV path decoding, and row transitions for final AQE exchanges ([#14793](https://github.com/NVIDIA/spark-rapids/pull/14793), [#14792](https://github.com/NVIDIA/spark-rapids/pull/14792), [#14778](https://github.com/NVIDIA/spark-rapids/pull/14778), [#14914](https://github.com/NVIDIA/spark-rapids/pull/14914))
107+
108+
For a detailed list of changes, please refer to the
109+
[CHANGELOG](https://github.com/NVIDIA/spark-rapids/blob/main/CHANGELOG.md).
110+
8111
## Release v26.04.2
9112
### Hardware Requirements:
10113

0 commit comments

Comments
 (0)