Skip to content
Merged
Show file tree
Hide file tree
Changes from 19 commits
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
bc855e4
Update documentation for cuDF for Spark rename
Jun 17, 2026
660c598
Merge branch 'main' of https://github.com/nvidia/cudf-spark into cudf…
Jun 17, 2026
f9398cd
Update generated docs
Jun 22, 2026
6c8ee35
addressing amahussein's comments
Jun 23, 2026
49395e9
Update naming in microk8s.md
Jun 23, 2026
ac97f11
Update license
Jun 23, 2026
f16a9f6
Prepend the first mention "NVIDIA", and other minor fixes
Jun 24, 2026
b9507b3
Merge branch 'main' of https://github.com/nvidia/cudf-spark into cudf…
Jul 25, 2026
e2066d1
Merge branch 'main' of https://github.com/nvidia/cudf-spark into cudf…
Jul 25, 2026
207c83a
1. Updated based on new naming guidance: New usage in heading and fir…
Jul 27, 2026
593ee5d
Merge branch 'main' of https://github.com/nvidia/cudf-spark into cudf…
Jul 27, 2026
b6676f1
Update contact us email
Jul 27, 2026
f85c6ad
Updated based on nvliyuan's feedback
Jul 27, 2026
bb1738b
Merge branch 'release/26.08' into cudf-spark-rename-1
sameerz Jul 27, 2026
72cf2d1
Updated based on GaryShen2008's feedback
Jul 28, 2026
0f3ee92
Merge branch 'cudf-spark-rename-1' of https://github.com/sameerz/spar…
Jul 28, 2026
a8f87d8
Merge branch 'release/26.08' into cudf-spark-rename-1
sameerz Jul 28, 2026
a07f09f
Merge branch 'release/26.08' of https://github.com/nvidia/cudf-spark …
Jul 28, 2026
5b8c88a
Merge branch 'cudf-spark-rename-1' of https://github.com/sameerz/spar…
Jul 28, 2026
7560c31
Update docs/dev/get-json-object-dump-tool.md
GaryShen2008 Jul 28, 2026
26349e6
Update CONTRIBUTING.md
GaryShen2008 Jul 28, 2026
1bc4c80
Update docs/dev/get-json-object-dump-tool.md
GaryShen2008 Jul 28, 2026
788a18a
Merge branch 'release/26.08' into cudf-spark-rename-1
GaryShen2008 Jul 28, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/ISSUE_TEMPLATE/bug_report.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
name: Bug report
about: Create a bug report to help us improve RAPIDS Accelerator for Apache Spark
about: Create a bug report to help us improve the NVIDIA cuDF plugin for Apache Spark
title: "[BUG]"
labels: "? - Needs Triage, bug"
assignees: ''
Expand Down
2 changes: 1 addition & 1 deletion .github/ISSUE_TEMPLATE/feature_request.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
name: Feature request
about: Suggest an idea for RAPIDS Accelerator for Apache Spark
about: Suggest an idea for the NVIDIA cuDF plugin for Apache Spark
title: "[FEA]"
labels: "? - Needs Triage, feature request"
assignees: ''
Expand Down
2 changes: 1 addition & 1 deletion .github/ISSUE_TEMPLATE/submit-question.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
name: Submit question
about: Ask a general question about RAPIDS Accelerator for Apache Spark here, or open a thread in the Discussions tab
about: Ask a general question about the NVIDIA cuDF plugin for Apache Spark here, or open a thread in the Discussions tab
title: "[QST]"
labels: "? - Needs Triage, question"
assignees: ''
Expand Down
4 changes: 2 additions & 2 deletions .github/PULL_REQUEST_TEMPLATE.md
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
<!--

Thank you for contributing to RAPIDS Accelerator for Apache Spark!
Thank you for contributing to the NVIDIA cuDF plugin for Apache Spark!

Please read https://github.com/NVIDIA/spark-rapids/blob/HEAD/CONTRIBUTING.md#creating-a-pull-request before making this PR.
Please read https://github.com/NVIDIA/cudf-spark/blob/HEAD/CONTRIBUTING.md#creating-a-pull-request before making this PR.

The following are the guidelines to help the review process go smoothly. Please read them carefully and fill out relevant information as much as possible.

Expand Down
6 changes: 3 additions & 3 deletions .github/copilot-instructions.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# GitHub Copilot Review Instructions for spark-rapids

spark-rapids is a GPU acceleration plugin for Apache Spark.
cudf-spark is a GPU acceleration plugin for Apache Spark.
For full project context, coding conventions, and code examples,
see [AGENTS.md](../AGENTS.md).

Expand All @@ -10,7 +10,7 @@ When reviewing shim code or GPU operator implementations, cross-reference
these upstream repositories to verify correctness:
- **apache/spark** — verify GPU behavior matches the CPU implementation in the target Spark version (expression evaluation semantics, null handling, type coercion, catalog/partition behavior)
- **rapidsai/cudf** — verify cuDF API usage (column operations, memory allocation semantics, null handling)
- **NVIDIA/spark-rapids-jni** — verify JNI binding signatures, memory ownership rules, error codes
- **NVIDIA/cudf-spark-jni** — verify JNI binding signatures, memory ownership rules, error codes

## Review Checklist

Expand All @@ -26,7 +26,7 @@ these upstream repositories to verify correctness:
- [ ] H5: Configuration — new RapidsConf without docs/defaults; should use .internal() if not user-visible; new features default off
- [ ] H6: Magic numbers — unexplained numeric literals without named constants or comments
- [ ] H7: Pre-merge CI gaps — only selected shims run unit tests; [databricks] needed for DB CI; feature-gated tests need explicit enable; limited Scala 2.13 coverage
- [ ] H8: Upstream dependencies — SNAPSHOT changes from spark-rapids-jni/cudf may break; verify API usage against upstream repos
- [ ] H8: Upstream dependencies — SNAPSHOT changes from cudf-spark-jni/cudf may break; verify API usage against upstream repos

## Do NOT Comment On

Expand Down
2 changes: 1 addition & 1 deletion CODE_OF_CONDUCT.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

## Overview

Define the code of conduct followed and enforced by the RAPIDS Accelerator for Apache Spark project
Define the code of conduct followed and enforced by the NVIDIA cuDF plugin for Apache Spark project

### Intended audience

Expand Down
32 changes: 16 additions & 16 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
@@ -1,15 +1,15 @@
# Contributing to RAPIDS Accelerator for Apache Spark
# Contributing to the NVIDIA cuDF plugin for Apache Spark

Contributions to RAPIDS Accelerator for Apache Spark fall into the following three categories.
Contributions to the NVIDIA cuDF plugin for Apache Spark fall into the following three categories.

1. To report a bug, request a new feature, or report a problem with
documentation, please file an [issue](https://github.com/NVIDIA/spark-rapids/issues/new/choose)
documentation, please file an [issue](https://github.com/NVIDIA/cudf-spark/issues/new/choose)
describing in detail the problem or new feature. The project team evaluates
and triages issues, and schedules them for a release. If you believe the
issue needs priority attention, please comment on the issue to notify the
team.
2. To propose and implement a new Feature, please file a new feature request
[issue](https://github.com/NVIDIA/spark-rapids/issues/new/choose). Describe the
[issue](https://github.com/NVIDIA/cudf-spark/issues/new/choose). Describe the
intended feature and discuss the design and implementation with the team and
community. Once the team agrees that the plan looks good, go ahead and
implement it using the [code contributions](#code-contributions) guide below.
Expand Down Expand Up @@ -48,7 +48,7 @@ building at least running to the `verify` phase, e.g.:
mvn verify
```

After a successful build, the RAPIDS Accelerator jar will be in the `dist/target/` directory.
After a successful build, the cudf plugin jar will be in the `dist/target/` directory.
Comment thread
GaryShen2008 marked this conversation as resolved.
Outdated
This will build the plugin for a single version of Spark. By default, this is Apache Spark
3.3.0. To build against other versions of Spark you use the `-Dbuildver=XXX` command line option
to Maven. For instance to build Spark 3.3.0 you would use:
Expand Down Expand Up @@ -83,7 +83,7 @@ build versions. See the next section for more details.

You might see a warning during scala-maven-plugin compile goal invocation.
```
[INFO] Compiling 94 Scala sources and 1 Java source to /home/user/gits/NVIDIA/spark-rapids/tests/target/spark3XY/test-classes ...
[INFO] Compiling 94 Scala sources and 1 Java source to /home/user/gits/NVIDIA/cudf-spark/tests/target/spark3XY/test-classes ...
OpenJDK 64-Bit Server VM warning: CodeCache is full. Compiler has been disabled.
OpenJDK 64-Bit Server VM warning: Try increasing the code cache size using -XX:ReservedCodeCacheSize=
CodeCache: size=245760Kb used=236139Kb max_used=243799Kb free=9620Kb
Expand Down Expand Up @@ -167,7 +167,7 @@ In many situations the user knows that the Plugin jar will be deployed for a sin
release. It is most commonly the case when a container image for a cloud or local deployment includes
Spark binaries as well. In such a case it is advantageous to create a jar with
a conventional class directory structure avoiding complications such as
[#3704](https://github.com/NVIDIA/spark-rapids/issues/3704). To this end add
[#3704](https://github.com/NVIDIA/cudf-spark/issues/3704). To this end add
`-DallowConventionalDistJar=true` when invoking Maven.

```bash
Expand Down Expand Up @@ -214,15 +214,15 @@ for a single Spark version Shim alone.
To this end in a pre-production build you can set the Boolean property
`dist.jar.compress` to `false`, its default value is `true`.

Furthermore, after the first build execution on the clean repository the spark-rapids-jni
Furthermore, after the first build execution on the clean repository the cudf-spark-jni
SNAPSHOT dependency typically does not change until the next nightly CI build, or the next install
to the local Maven repo if you are working on a change to the native code. So you can save
significant time spent on repeated unpacking these dependencies by adding `-Drapids.jni.unpack.skip`
to the `dist` build command.

The time saved is more significant if you are merely changing
the `aggregator` module, or the `dist` module, or just incorporating changes from
[spark-rapids-jni](https://github.com/NVIDIA/spark-rapids-jni/blob/branch-23.04/CONTRIBUTING.md#local-testing-of-cross-repo-contributions-cudf-spark-rapids-jni-and-spark-rapids)
[cudf-spark-jni](https://github.com/NVIDIA/cudf-spark-jni/blob/branch-23.04/CONTRIBUTING.md#local-testing-of-cross-repo-contributions-cudf-spark-rapids-jni-and-spark-rapids)

For example, to quickly repackage `rapids-4-spark` after the
initial `./build/buildall` you can iterate by invoking
Expand Down Expand Up @@ -280,7 +280,7 @@ Last tested with IntelliJ IDEA 2023.1.2 (Community Edition)

##### Manual Maven Install for a target Spark build

Before proceeding with importing spark-rapids into IDEA or switching to a different Spark release
Before proceeding with importing cudf-spark into IDEA or switching to a different Spark release
profile, execute the install phase with the corresponding `buildver`, e.g. for Spark 3.4.0:

```bash
Expand Down Expand Up @@ -378,7 +378,7 @@ rm -vf .bloop
ln -s .bloop-spark330 .bloop
```

You can now open the spark-rapids as a
You can now open the cudf-spark as a
[BSP project in IDEA](https://www.jetbrains.com/help/idea/bsp-support.html)

Read on for VS Code Scala Metals instructions.
Expand Down Expand Up @@ -446,15 +446,15 @@ We welcome pull requests with tips on how to setup your favorite IDE!

### Your first issue

1. Read the [Developer Overview](docs/dev/README.md) to understand how the RAPIDS Accelerator
1. Read the [Developer Overview](docs/dev/README.md) to understand how the cuDF
plugin works.
2. Find an issue to work on. The best way is to look for the
[good first issue](https://github.com/NVIDIA/spark-rapids/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22)
or [help wanted](https://github.com/NVIDIA/spark-rapids/issues?q=is%3Aissue+is%3Aopen+label%3A%22help+wanted%22)
[good first issue](https://github.com/NVIDIA/cudf-spark/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22)
or [help wanted](https://github.com/NVIDIA/cudf-spark/issues?q=is%3Aissue+is%3Aopen+label%3A%22help+wanted%22)
labels.
3. Comment on the issue stating that you are going to work on it.
4. Code! Make sure to update unit tests and integration tests if needed! [refer to test section](#testing-your-code)
5. When done, [create your pull request](https://github.com/NVIDIA/spark-rapids/compare).
5. When done, [create your pull request](https://github.com/NVIDIA/cudf-spark/compare).
6. Verify that CI passes all [status checks](https://help.github.com/articles/about-status-checks/).
Fix if needed.
7. Wait for other developers to review your code and update code as needed.
Expand All @@ -464,7 +464,7 @@ Remember, if you are unsure about anything, don't hesitate to comment on issues
and ask for clarifications!

### Code Formatting
RAPIDS Accelerator for Apache Spark follows the same coding style guidelines as the Apache Spark
The NVIDIA cuDF plugin for Apache Spark follows the same coding style guidelines as the Apache Spark
project. For IntelliJ IDEA users, an
[example code style settings file](docs/dev/idea-code-style-settings.xml) is available in the
`docs/dev/` directory.
Expand Down
6 changes: 3 additions & 3 deletions DF_UDF_README.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,21 +7,21 @@ are also used as a way to standardize processing logic across an organization or
But UDFs come with some downsides. The biggest one is visibility into the processing being done. SQL is a language that
can be highly optimized. But a UDF in most cases is a black box, that the SQL optimizer cannot do anything about.
This can result in less than ideal query planning. Additionally, accelerated execution environments, like the
RAPIDS Accelerator for Apache Spark have no easy way to replace UDFs with accelerated versions, which can result in
NVIDIA cuDF plugin for Apache Spark have no easy way to replace UDFs with accelerated versions, which can result in
slow performance.

This attempts to add visibility to the code reuse use case by providing a way to implement a UDF in terms of dataframe
commands.

## Setup

The dataframe UDF plugin is packaged in the same jar as the RAPIDS Accelerator for Apache Spark. This jar will need to
The dataframe UDF plugin is packaged in the same jar as the cuDF plugin. This jar will need to
be added as a compile time dependency for code that wants to use this feature as well as adding the jar to your Spark
classpath just like you would do for GPU acceleration.

If you plan to not use the GPU accelerated processing, but still want dataframe UDF support on CPU applications then
add `com.nvidia.spark.DFUDFPlugin` to the `spark.sql.extensions` config. If you do use GPU accelerated processing
the RAPIDS Plugin will enable this automatically. You don't need to set the `spark.sql.extensions` config, but it
the cuDF plugin will enable this automatically. You don't need to set the `spark.sql.extensions` config, but it
won't hurt anything if you do add it. Now you can implement a UDF in terms of Dataframe operations.

## Usage
Expand Down
29 changes: 16 additions & 13 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,15 +1,19 @@
# RAPIDS Accelerator For Apache Spark
NOTE: For the latest stable [README.md](https://github.com/nvidia/spark-rapids/blob/main/README.md) ensure you are on the main branch.
# NVIDIA cuDF plugin for Apache Spark
NOTE: For the latest stable [README.md](https://github.com/nvidia/cudf-spark/blob/main/README.md) ensure you are on the main branch.

The RAPIDS Accelerator for Apache Spark provides a set of plugins for
[Apache Spark](https://spark.apache.org) that leverage GPUs to accelerate processing
via the [RAPIDS](https://rapids.ai) libraries.
The NVIDIA cuDF plugin for [Apache Spark](https://spark.apache.org) provides a plugin library that
leverages GPUs to accelerate processing via the [cuDF](https://github.com/rapidsai/cudf) (CUDA
DataFrame) libraries.

Documentation on the current release can be found [here](https://nvidia.github.io/cudf-spark/).

To get started and try the plugin out use the [getting started guide](https://docs.nvidia.com/spark-rapids/user-guide/latest/getting-started/overview.html).

[![Ask DeepWiki](https://deepwiki.com/badge.svg)](https://deepwiki.com/NVIDIA/spark-rapids)
[![Ask DeepWiki](https://deepwiki.com/badge.svg)](https://deepwiki.com/NVIDIA/cudf-spark)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: The deepwiki page still says RAPIDS Accelerator for Apache Spark. Are we able to update it accordingly?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Once we rename the link it should trigger a rebuild of the deepwiki page. Let's see if it works.


Note: The NVIDIA cuDF plugin for Apache Spark was formerly known as the RAPIDS Accelerator for
Apache Spark. The RAPIDS name will be sunset over time. Github links from `spark-rapids` will
redirect to `cudf-spark`. Artifact names will remain the same for now.

## Compatibility

Expand All @@ -29,8 +33,8 @@ The plugin has a set of Spark configs that control its behavior and are document
## Issues & Questions

We use github to track bugs, feature requests, and answer questions. File an
[issue](https://github.com/NVIDIA/spark-rapids/issues/new/choose) for a bug or feature request. Ask
or answer a question on the [discussion board](https://github.com/NVIDIA/spark-rapids/discussions).
[issue](https://github.com/NVIDIA/cudf-spark/issues/new/choose) for a bug or feature request. Ask
or answer a question on the [discussion board](https://github.com/NVIDIA/cudf-spark/discussions).

## Download

Expand All @@ -46,8 +50,8 @@ See the [build instructions in the contributing guide](CONTRIBUTING.md#building-
Tests are described [here](tests/README.md).

## Integration
The RAPIDS Accelerator For Apache Spark does provide some APIs for doing zero copy data
transfer into other GPU enabled applications. It is described
The cuDF plugin provides some APIs for doing zero copy data transfer into other GPU enabled
applications. It is described
[here](https://docs.nvidia.com/spark-rapids/user-guide/latest/additional-functionality/ml-integration.html).

Currently, we are working with XGBoost to try to provide this integration out of the box.
Expand All @@ -67,9 +71,8 @@ for more details on how to use the tools.

## Dependency for External Projects

If you need to develop some functionality on top of RAPIDS Accelerator For Apache Spark (we currently
limit support to GPU-accelerated UDFs) we recommend you declare our distribution artifact
as a `provided` dependency.
If you develop functionality on top of the cuDF plugin (we currently limit support to
GPU-accelerated UDFs) we recommend you declare the distribution artifact as a `provided` dependency.

```xml
<dependency>
Expand Down
13 changes: 6 additions & 7 deletions datagen/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,12 +8,11 @@ data between tables for joins. To accomplish this we wrote

## Setup Environment

To get started with big data generation the first thing you need to do is
to include the appropriate jar on the classpath for your version of Apache Spark.
Note that this does not run on the GPU, but it does use parts of the shim framework
that the RAPIDS Accelerator does. The jar is specific to the version of Spark you
are using and is not pushed to Maven Central. Because of this you will have to
build it from source yourself.
To get started with big data generation the first thing you need to do is to include the appropriate
jar on the classpath for your version of Apache Spark. Note that this does not run on the GPU, but
it does use parts of the shim framework that the NVIDIA cuDF plugin for Apache Spark does. The jar
is specific to the version of Spark you are using and is not pushed to Maven Central. Because of
this you will have to build it from source yourself.

```shell
cd datagen
Expand Down Expand Up @@ -344,7 +343,7 @@ spark.time(fdf.join(ddf).groupBy("agg_key").agg(min("value"),
Time taken: 890163 ms
```

Or you could run it with the RAPIDS Accelerator where the data generations is still
Or you could run it with the cuDF plugin where the data generations is still
done on the CPU and cut the time down to just 75,696 ms

But what if we wanted to join on multiple columns, or do an aggregation with
Expand Down
2 changes: 1 addition & 1 deletion delta-lake/README.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# RAPIDS Accelerator for Apache Spark Delta Lake Support
# NVIDIA cuDF plugin for Apache Spark Delta Lake Support

The Delta Lake support is organized into multiple Maven projects, one per Delta Lake minor
version that is supported. This allows each submodule to build against the Delta Lake minor
Expand Down
4 changes: 2 additions & 2 deletions dist/README.md
Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@
---
layout: page
title: RAPIDS Accelerator for Apache Spark Distribution Packaging
title: NVIDIA cuDF plugin for Apache Spark Distribution Packaging
nav_order: 1
parent: Developer Overview
---
# RAPIDS Accelerator for Apache Spark Distribution Packaging
# NVIDIA cuDF plugin for Apache Spark Distribution Packaging

The distribution module creates a jar with support for the Spark versions you need combined into a single jar.

Expand Down
8 changes: 4 additions & 4 deletions docs/_config.yml
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
#
# Copyright (c) 2020, NVIDIA CORPORATION.
# Copyright (c) 2020-2026, NVIDIA CORPORATION.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
Expand All @@ -18,10 +18,10 @@
remote_theme: pmarsceill/just-the-docs

aux_links:
"RAPIDS Accelerator for Apache Spark User Guide":
"NVIDIA cuDF plugin for Apache Spark User Guide":
- "//docs.nvidia.com/spark-rapids/user-guide/latest/index.html"
"RAPIDS Accelerator for Apache Spark Plugin on Github":
- "//github.com/nvidia/spark-rapids"
"NVIDIA cuDF plugin for Apache Spark on Github":
- "//github.com/nvidia/cudf-spark"

plugins:
- jekyll-optional-front-matter # GitHub Pages
Expand Down
Loading
Loading