obj: add generic GPU-direct S3-over-RDMA path (gds=true)

Add a vendor-neutral S3-over-RDMA data path to the object plugin's standard
S3 client, implementing the published `x-amz-rdma-*` protocol (NVIDIA cuObject /
aws-c-s3 convention). Any compliant endpoint (MinIO AIStor today, and AWS S3 if
it adopts it) works with no per-vendor code.

What it does:
- New s3/rdma_protocol.h (dependency-free wire helpers; unit tested) and
  s3/rdma.{h,cpp} (cuObject token API: cuMemObjGetRDMAToken/PutRDMAToken, plus
  a SigV4 UNSIGNED-PAYLOAD control plane that carries x-amz-rdma-token and reads
  x-amz-rdma-reply / x-amz-rdma-bytes-transferred).
- RDMA is an explicit opt-in via the `gds=true` backend param. The standard
  client attempts RDMA for registered DRAM/VRAM buffers; VRAM (GPU-direct) is
  advertised only under gds + an available fabric.
- No automatic HTTP fallback: an RDMA decline/failure under gds is a hard error.
  Auto-fallback relies on the server returning x-amz-rdma-reply:501 for
  unsupported RDMA; a server that silently ignores the token would accept a
  body-less PUT as a 0-byte object. Until that contract is universal we require
  the explicit opt-in. The future fallback path is documented in code/README.
- Fail-safe PUT: success is reported only when the server returns
  x-amz-rdma-reply:200/204, so a misconfigured (non-RDMA) endpoint errors
  instead of silently writing a 0-byte object.

Why not s3_accel/dell: the legacy Dell engine uses a non-standard x-rdma-info
header, a static whole-backend engine selection, and a PUT that does not check
any reply header. It is kept for backward compatibility (accelerated=true,
type=dell) but is deprecated and superseded by this path; see
s3_accel/TODO.md for the removal plan.

Docs: RDMA_PROTOCOL.md documents the server-side wire contract and a vendor
compliance checklist; README gains a GPU-Direct section and marks
accelerated/type as deprecated. nixl_object_test gains -g/--gds (and -v works
with --gds); -a/--accelerated is marked deprecated.

Note: the AWS SDK control-plane code in s3/rdma.cpp compiles only with
cuobjclient + aws-sdk-cpp present and is validated on RDMA hardware; the
protocol layer is covered by a unit test (test/gtest/unit/obj/rdma_protocol.cpp).

Signed-off-by: Harshavardhana <harsha@minio.io>

Author: harshavardhana

src/plugins/obj/RDMA_PROTOCOL.md

@@ -0,0 +1,120 @@
+<!--
+SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+SPDX-License-Identifier: Apache-2.0
+-->
+
+# S3-over-RDMA Wire Protocol
+
+This document specifies the GPU-direct S3-over-RDMA protocol the object plugin
+speaks, so that **any S3-compatible storage vendor can comply on the server side
+without NIXL-specific code**. It is a vendor-neutral AWS S3 convention (the same
+one used by NVIDIA cuObject / `aws-c-s3`): the control plane stays standard
+HTTP+SigV4, while the object payload moves out-of-band over RDMA. A server that
+implements the contract below works with this client when the caller opts in via
+`gds=true`.
+
+> The client treats RDMA as an explicit opt-in and does **not** auto-fall-back to
+> HTTP today (see the README's "No automatic fallback (yet)"). A compliant server
+> signals an unsupported request with `x-amz-rdma-reply: 501`, but a
+> *non*-compliant server may silently ignore the token and accept a body-less PUT
+> as a 0-byte object — so auto-fallback becomes safe only once the `501` contract
+> below is universal.
+
+The reference server implementation is MinIO AIStor (`internal/rdma`, repo at
+`../aistor`).
+
+## Transport
+
+- RDMA **Dynamically Connected (DC)** transport over **InfiniBand** or
+  **RoCEv2** (cuObject protocol `CUOBJ_PROTO_RDMA_DC_V1`).
+- **GET**: the server issues `RDMA_WRITE`, pushing object bytes directly into the
+  client's registered buffer (host DRAM or GPU VRAM).
+- **PUT** / **UploadPart**: the server issues `RDMA_READ`, pulling bytes directly
+  out of the client's registered buffer.
+- The HTTP request/response body is **empty**; payload never traverses it.
+
+## Headers
+
+| Header | Direction | Meaning |
+|--------|-----------|---------|
+| `x-amz-rdma-token` | request | RDMA negotiation token (see format below). Its presence is what marks a request as RDMA. |
+| `x-amz-content-sha256: UNSIGNED-PAYLOAD` | request | Body is not signed (there is no body). |
+| `Content-Length: 0` | request (PUT) | No HTTP body; data is delivered via `RDMA_READ`. |
+| `Range: bytes=a-b` | request (GET) | Optional; selects a byte range (server replies `206`). |
+| `x-amz-checksum-crc64nvme` | request/response | Optional CRC64NVME checksum (PUT/UploadPart). |
+| `x-amz-rdma-reply` | response | Outcome marker. **Required on success.** `200`/`204`/`206` = success; `501` (or absent) = server declined RDMA. Under `gds=true` the client treats a decline as a hard error (no automatic HTTP fallback today — see README). |
+| `x-amz-rdma-bytes-transferred` | response (GET) | Actual bytes moved over RDMA (may be `< requested` for ranged GETs). |
+| `ETag`, `x-amz-checksum-crc64nvme` | response (PUT) | Standard S3 result metadata. |
+
+### Token format
+
+```text
+x-amz-rdma-token: <descriptor>:<start_addr_hex>:<size_hex>
+```
+
+- `<descriptor>` — opaque, fixed-length RDMA memory descriptor minted by the
+  client's RDMA provider (cuObject). The server forwards it to its RDMA provider
+  (cuObjServer); it must not interpret the bytes.
+- `<start_addr_hex>` — client buffer start address, 16-digit lowercase hex.
+- `<size_hex>` — transfer size in bytes, 16-digit lowercase hex.
+
+The two trailing fields are `:`-separated and base-16; the server splits them off
+the end and treats the remaining prefix as the descriptor. Trailing `;` (some
+clients append one) must be tolerated.
+
+## Request flow (server obligations)
+
+1. **Authenticate** the request with standard SigV4. `x-amz-rdma-token` is part
+   of the signed headers; `x-amz-content-sha256` is `UNSIGNED-PAYLOAD`.
+2. **Detect** an RDMA request by the presence of `x-amz-rdma-token`; parse it
+   into `(descriptor, start_addr, size)`.
+3. **Decide** whether RDMA can be honored (fabric reachable, buffer registrable,
+   object/permissions valid).
+   - **If not** → respond `x-amz-rdma-reply: 501`. You MAY include a normal HTTP
+     error body (do not force `Content-Length: 0`). The `501` signal is what lets
+     a client safely decide what to do; today this client treats it as an error
+     under `gds=true` (it does not auto-retry over HTTP — see README), but the
+     signal is required so future clients can fall back transparently.
+4. **Transfer** using the cuObject server APIs (register a local buffer, then):
+   - GET / RANGE_GET → `RDMA_WRITE` the requested bytes into the client buffer.
+   - PUT / UploadPart → `RDMA_READ` the bytes from the client buffer and persist
+     them as the object/part.
+5. **Respond** on success:
+   - `x-amz-rdma-reply: 200` (full GET/PUT), `204` (PUT, no content), or `206`
+     (ranged GET partial).
+   - `x-amz-rdma-bytes-transferred: <n>` for GET.
+   - `Content-Length: 0` and **no HTTP body** (the data went over RDMA — without
+     this the client blocks waiting for body bytes that never arrive).
+   - `ETag` (and `x-amz-checksum-crc64nvme` if requested) for PUT.
+
+## Fail-safe contract (why the reply header is mandatory on success)
+
+A compliant PUT carries `Content-Length: 0` and no body. A server that does
+**not** implement this protocol would accept that request as a *legitimate
+zero-byte object* and return `200 OK` + an `ETag`. To prevent that silent data
+loss, the client treats a PUT as successful **only** when `x-amz-rdma-reply` is
+present and `200`/`204`. Therefore a compliant server MUST set
+`x-amz-rdma-reply` on every successful RDMA response. (GET is inherently safe:
+an absent reply header is read as "declined".)
+
+## Multipart uploads
+
+UploadPart uses the same body-less, token-carrying PUT with the standard
+`?uploadId=<id>&partNumber=<n>` query parameters (`1 ≤ n ≤ 10000`). The part's
+`ETag`/checksum are returned as usual; CompleteMultipartUpload is an ordinary
+HTTP call.
+
+## Compliance checklist
+
+- [ ] Parse `x-amz-rdma-token` as `<descriptor>:<addr16>:<size16>` (tolerate a
+      trailing `;`).
+- [ ] Authenticate with SigV4 treating the body as `UNSIGNED-PAYLOAD`.
+- [ ] GET ⇒ `RDMA_WRITE` into the client buffer; PUT ⇒ `RDMA_READ` from it.
+- [ ] Always set `x-amz-rdma-reply` (`200`/`204`/`206`) on success, `501` to decline.
+- [ ] Set `x-amz-rdma-bytes-transferred` on GET; force `Content-Length: 0` with
+      no body on every success.
+- [ ] Return `ETag`/checksum for PUT; honor `Range` for GET (`206`).
+- [ ] Fall through to normal HTTP semantics for non-RDMA requests.
+
+A vendor that satisfies this checklist is supported by the NIXL object plugin's
+default S3 client automatically — no NIXL-side engine or plugin is required.

src/plugins/obj/README.md

@@ -70,8 +70,9 @@ Backend parameters are passed as a key-value map (`nixl_b_params_t`) when creati
 | `resp_checksum` | Response checksum validation (`required`/`supported`) | - | No |
 | `ca_bundle` | path to a custom certificate bundle | - | No |
 | `crtMinLimit` | Minimum object size (bytes) to use S3 CRT client for high-performance transfers | Disabled**** | No |
-| `accelerated` | Enable S3 Accelerated engine (`true`/`false`) | `false` | No |
-| `type` | Accelerated engine type (`dell`, etc.) | - | No |
+| `gds` | Enable GPU-direct S3-over-RDMA on the standard client (`true`/`false`). Opt-in; requires an endpoint that implements the protocol. See [GPU-Direct (S3 over RDMA)](#gpu-direct-s3-over-rdma). | `false` | No |
+| `accelerated` | **(Deprecated/Legacy)** Enable a registry-provided vendor accel engine (`true`/`false`). Superseded by `gds`; kept for legacy Dell. | `false` | No |
+| `type` | **(Deprecated/Legacy)** Vendor accel engine type (`dell`). Superseded by `gds`. | - | No |
 
 \* If `access_key` and `secret_key` are not provided, the AWS SDK will attempt to use default credential providers (IAM roles, environment variables, credential files, etc.)
 
@@ -215,6 +216,102 @@ This configuration automatically uses the high-performance S3 CRT client for obj
 
 > **Part size and the 5 MiB floor**: AWS S3 requires a minimum part size of 5 MiB for all parts except the last. If `crtMinLimit` is set below 5 MiB, the CRT SDK silently clamps the part size to 5 MiB (logging a warning) while still triggering multipart at `crtMinLimit`. Objects smaller than 5 MiB are uploaded as a single-part multipart request, which S3 permits. The 5 MiB minimum is an AWS service constraint and cannot be bypassed. To avoid the silent clamp, use `crtMinLimit >= 5242880` (5 MiB).
 
+## GPU-Direct (S3 over RDMA)
+
+When built against the `cuobjclient` library, run on a host with an RDMA fabric
+(InfiniBand or RoCEv2), and **explicitly enabled with `gds=true`**, the standard
+S3 client accelerates GET/PUT using **GPU-direct S3 over RDMA**. This is the
+idiomatic path; enable it only against an endpoint you know implements the
+protocol:
+
+```cpp
+nixl_b_params_t params = {
+    {"bucket", "models"}, {"endpoint_override", "http://aistor:9000"},
+    {"gds", "true"}
+};
+agent.createBackend("obj", params);
+```
+
+> **Why opt-in and not auto-detected?** See
+> [No automatic fallback (yet)](#no-automatic-fallback-yet) — auto-probing is
+> unsafe until servers reliably signal `501` for unsupported RDMA.
+
+### How it works
+
+The implementation follows the published, vendor-neutral S3 RDMA convention
+(NVIDIA cuObject / `aws-c-s3`), so it works against any compliant endpoint
+(MinIO AIStor today; AWS S3 transparently if it ever adopts it):
+
+1. On `registerMem`, DRAM/VRAM buffers are pinned for RDMA via cuObject
+   (`cuMemObjGetDescriptor`). VRAM requires this; DRAM registration is
+   best-effort.
+2. On transfer, the client mints a per-op RDMA token and issues a **body-less**,
+   SigV4 `UNSIGNED-PAYLOAD` GET/PUT carrying `x-amz-rdma-token:
+   <descriptor>:<addr>:<size>`.
+3. The server performs an `RDMA_WRITE` (GET) or `RDMA_READ` (PUT) **directly
+   to/from the registered buffer** — including GPU VRAM — bypassing the host CPU
+   and the HTTP body.
+4. The server confirms with `x-amz-rdma-reply: 200/204/206` and reports the
+   moved byte count via `x-amz-rdma-bytes-transferred`.
+
+### No automatic fallback (yet)
+
+RDMA is an **explicit opt-in** (`gds=true`), not an auto-probe, and when enabled
+an RDMA decline/failure is a **hard error** — the client does *not* silently fall
+back to HTTP. This is deliberate:
+
+- The fallback handshake relies on the server returning `x-amz-rdma-reply: 501`
+  (or omitting the reply header) when it cannot honor RDMA. A server that instead
+  **silently ignores** `x-amz-rdma-token` would accept our `Content-Length: 0`
+  body-less PUT as a legitimate **zero-byte object** and return `200`+ETag —
+  silent data loss. The client cannot safely distinguish that case by probing.
+- So the caller asserts RDMA support via `gds=true`; if the endpoint then
+  declines, that is a misconfiguration worth surfacing, not papering over.
+
+> **Fail-safe PUT:** even under `gds=true`, a PUT is reported successful **only**
+> when the server returns `x-amz-rdma-reply: 200/204`. A bare HTTP `200` is never
+> trusted, so a mistaken endpoint surfaces as an error rather than a silent
+> zero-byte write. (GET is inherently safe: an absent reply header reads as a
+> decline.)
+>
+> **FUTURE — transparent fallback.** Once server implementations universally
+> return `x-amz-rdma-reply: 501` for unsupported RDMA, the client can safely
+> auto-detect RDMA and fall back to HTTP for host (DRAM) memory — making `gds`
+> unnecessary. The building blocks (reply parsing, the synchronous HTTP path) are
+> in place; the fallback is intentionally disabled until that contract is
+> universal. VRAM can never fall back (the SDK cannot stream a GPU pointer).
+
+### Why this supersedes the legacy `s3_accel` vendor engines
+
+The earlier `s3_accel/` path (e.g. `s3_accel/dell`, selected via
+`accelerated=true, type=dell`) is **retained only for backward compatibility**.
+It is not the recommended approach and is protocol-incorrect for general use:
+
+- It uses a **non-standard `x-rdma-info`** header instead of the established
+  `x-amz-rdma-token` / `x-amz-rdma-reply` convention, so it only works against
+  one vendor's server.
+- It sends `Content-Length: 0` PUTs **without checking any RDMA reply header**,
+  so against a non-RDMA endpoint it silently writes 0-byte objects.
+- It has **no fallback**: RDMA selection is a static, whole-backend choice made
+  at `createBackend` time, not a per-transfer negotiation.
+- It models RDMA as a **separate vendor engine** behind a registry, requiring
+  bespoke per-vendor code, when RDMA is really just a transparent client
+  capability.
+
+The generic path above fixes all four: it is a per-transfer capability of the
+standard client, follows the public protocol, fails safely on a non-RDMA
+response instead of writing a 0-byte object (no silent fallback — see [No
+automatic fallback (yet)](#no-automatic-fallback-yet)), and needs **no per-vendor
+code** — any vendor that complies with the protocol is supported automatically.
+
+### Protocol specification (for storage vendors)
+
+The full wire contract — headers, token format, transport, server obligations,
+the fail-safe success rule, and a compliance checklist — is documented in
+[**RDMA_PROTOCOL.md**](RDMA_PROTOCOL.md). A storage vendor that implements that
+contract on its S3 server is supported by this client out of the box, with no
+NIXL-side changes.
+
 ## Transfer Operations
 
 The Object Storage backend supports read and write operations between local memory and S3 objects. Here are the key aspects of transfer operations:
@@ -259,7 +356,14 @@ The client selection happens transparently during transfer operations, requiring
 - The `checkXfer` function can be used to poll for operation completion
 - The request handle must be released using `releaseReqH` after the operation is complete
 
-## Extending with Vendor-Specific Implementations
+## Extending with Vendor-Specific Implementations (Legacy)
+
+> **⚠️ Legacy.** This vendor-engine + registry mechanism is retained for
+> backward compatibility only. For GPU-direct/RDMA acceleration, prefer the
+> built-in [GPU-Direct (S3 over RDMA)](#gpu-direct-s3-over-rdma) path, which is
+> transparent, protocol-compliant, and requires no per-vendor code. A new vendor
+> only needs to implement the standard `x-amz-rdma-*` protocol on its server to
+> be supported automatically; no NIXL-side engine is required.
 
 The object plugin uses a modular, inheritance-based architecture that makes it easy to add vendor-specific backends without modifying core engine logic.
 

src/plugins/obj/meson.build

@@ -24,6 +24,12 @@ obj_sources = [
     's3/client.h',
     's3/engine_impl.cpp',
     's3/engine_impl.h',
+    # Generic, vendor-neutral S3-over-RDMA (x-amz-rdma-*) data path. rdma.cpp is
+    # internally guarded by HAVE_CUOBJ_CLIENT and compiles to nothing when the
+    # cuObjClient library is absent; rdma_protocol.h is dependency-free.
+    's3/rdma_protocol.h',
+    's3/rdma.cpp',
+    's3/rdma.h',
     's3_crt/engine_impl.cpp',
     's3_crt/engine_impl.h',
     's3_crt/client.cpp',

src/plugins/obj/obj_backend.cpp

@@ -26,6 +26,24 @@
 // -----------------------------------------------------------------------------
 // Obj Engine Implementation
 // -----------------------------------------------------------------------------
+//
+// Engine selection:
+//   * Default (no params)         -> DefaultObjEngineImpl
+//   * crtMinLimit > 0             -> S3CrtObjEngineImpl
+//   * accelerated=true, type=...  -> registry-provided vendor engine (LEGACY)
+//
+// The idiomatic GPU-direct path is the S3-over-RDMA capability built into the
+// standard S3 client (see s3/rdma.h): it follows the published `x-amz-rdma-*`
+// protocol and requires NO per-vendor code — any S3 endpoint that implements the
+// protocol (e.g. MinIO AIStor) works, and so would AWS S3 if it adopted it. It is
+// an explicit opt-in via `gds=true` (not auto-probed, and no silent HTTP fallback
+// — see s3/client.cpp for the rationale and the future auto-fallback path).
+//
+// The `objAccelEngineRegistry` + `accelerated`/`type` path below is RETAINED ONLY
+// FOR LEGACY vendor engines (e.g. s3_accel/dell, which predates the protocol and
+// uses a non-standard `x-rdma-info` header). New deployments should use the
+// default engine; the generic client-level RDMA path supersedes the vendor
+// engines.
 
 namespace {
 

src/plugins/obj/obj_backend.h

@@ -85,6 +85,17 @@ class iS3Client {
      */
     virtual void
     checkObjectExistsAsync(std::string_view key, check_object_callback_t callback) = 0;
+
+    /**
+     * Whether this client's GPU-direct S3-over-RDMA fast path is fully ready
+     * (cuObject fabric connected AND control plane + executor available). The
+     * engine uses this to decide whether VRAM_SEG can be safely advertised:
+     * a GPU pointer must never reach the HTTP path. Defaults to false.
+     */
+    virtual bool
+    supportsRdma() const {
+        return false;
+    }
 };
 
 /**