Skip to content

Commit 96b0846

Browse files
harshavardhanaharshavardhana
committed
obj: add standard-protocol GPU-direct S3-over-RDMA (accelerated=true)
Add a vendor-neutral S3-over-RDMA data path to the object plugin, selected via the accelerated=true backend param with no type (or type=s3). It implements the published x-amz-rdma-* protocol (the NVIDIA cuObject / aws-c-s3 convention) and needs no per-vendor code; MinIO AIStor implements it today, and AWS S3 would work transparently if it adopts it. - New s3/rdma_protocol.h: dependency-free wire helpers (token formatting, reply parsing), covered by a unit test. - New s3/rdma.{h,cpp}: the cuObject RDMA token API plus a SigV4 UNSIGNED-PAYLOAD control plane that carries x-amz-rdma-token and reads x-amz-rdma-reply / x-amz-rdma-bytes-transferred. The object payload moves out-of-band over RDMA (server RDMA_READ on PUT, RDMA_WRITE on GET); the HTTP body is empty. - New s3_accel/generic/: the standard-S3 engine, registered in the accel registry under type "s3". accelerated=true with no type resolves to it; RDMA is a capability of the standard S3 client, enabled for this engine. It is the preferred GPU-direct path. - Explicit opt-in: an RDMA decline/failure is a hard error, with no automatic HTTP fallback. - RDMA_PROTOCOL.md documents the server-side wire contract and a vendor compliance checklist; the README gains a GPU-Direct section. - nixl_object_test gains -a/--accelerated (standard-S3) and keeps -T/--type for vendor engines. The Dell engine (accelerated=true, type=dell) is a vendor-specific engine for servers that speak Dell's x-rdma-info header. Validated end-to-end against MinIO AIStor over RoCEv2: host DRAM and GPU-direct VRAM PUT/GET with data verification. Signed-off-by: Harshavardhana <harsha@minio.io>
1 parent b775042 commit 96b0846

23 files changed

Lines changed: 1603 additions & 141 deletions

‎src/plugins/obj/RDMA_PROTOCOL.md‎

Lines changed: 152 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,152 @@
1+
<!--
2+
SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
3+
SPDX-License-Identifier: Apache-2.0
4+
-->
5+
6+
# S3-over-RDMA Wire Protocol
7+
8+
This document specifies the GPU-direct S3-over-RDMA protocol the object plugin
9+
speaks, so that **any S3-compatible storage vendor can comply on the server side
10+
without NIXL-specific code**. The control plane stays standard HTTP + SigV4,
11+
while the object payload moves out-of-band over RDMA. A server that implements
12+
the contract below works with this client when the caller opts in via
13+
`accelerated=true` (the standard-S3 engine; no `type`, or `type=s3`).
14+
15+
**Provenance.** The `x-amz-rdma-*` header names and the token layout come
16+
directly from NVIDIA's published [cuObject documentation][cuobj] (the
17+
client-side API and the RDMA-enabled GET/PUT workflow). The wire protocol was
18+
written up as a spec ([aws-c-s3 `RDMA_PROTOCOL_SPEC.md`][spec]) and proposed to
19+
AWS. It is not yet GA'ed by AWS; however AWS refined it inline with what they
20+
could eventually implement, so it is a stable, vendor-neutral base to
21+
standardize on rather than an endpoint-specific convention.
22+
23+
[cuobj]: https://docs.nvidia.com/gpudirect-storage/cuobject/index.html
24+
[spec]: https://github.com/KiranModukuri/aws-c-s3/blob/nvidia_rdma/RDMA_PROTOCOL_SPEC.md
25+
26+
The standard-S3 engine is the **preferred** GPU-direct path: it follows this
27+
proposed protocol and needs no per-vendor code, so it works against any
28+
conformant server. Vendor `type=...` engines exist only for servers that speak a
29+
vendor's own RDMA headers.
30+
31+
> The client treats RDMA as an explicit opt-in and does **not** auto-fall-back to
32+
> HTTP today (see the README's "No automatic fallback (yet)"). A compliant server
33+
> signals an unsupported request with `x-amz-rdma-reply: 501`, but a
34+
> _non_-compliant server may silently ignore the token and accept a body-less PUT
35+
> as a 0-byte object — so auto-fallback becomes safe only once the `501` contract
36+
> below is universal.
37+
38+
The reference server implementation is MinIO AIStor (`internal/rdma`, repo at
39+
`../aistor`).
40+
41+
## Transport
42+
43+
- RDMA **Dynamically Connected (DC)** transport over **InfiniBand** or
44+
**RoCEv2** (cuObject protocol `CUOBJ_PROTO_RDMA_DC_V1`).
45+
- **GET**: the server issues `RDMA_WRITE`, pushing object bytes directly into the
46+
client's registered buffer (host DRAM or GPU VRAM).
47+
- **PUT** / **UploadPart**: the server issues `RDMA_READ`, pulling bytes directly
48+
out of the client's registered buffer.
49+
- The HTTP request/response body is **empty**; payload never traverses it.
50+
51+
## Headers
52+
53+
| Header | Direction | Meaning |
54+
| ---------------------------------------- | ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
55+
| `x-amz-rdma-token` | request | RDMA negotiation token (see format below). Its presence is what marks a request as RDMA. |
56+
| `x-amz-content-sha256: UNSIGNED-PAYLOAD` | request | Body is not signed (there is no body). |
57+
| `Content-Length: 0` | request (PUT) | No HTTP body; data is delivered via `RDMA_READ`. |
58+
| `Range: bytes=a-b` | request (GET) | Optional; selects a byte range (server replies `206`). |
59+
| `x-amz-checksum-crc64nvme` | request/response | Optional CRC64NVME checksum (PUT/UploadPart). |
60+
| `x-amz-rdma-reply` | response | Outcome marker. **Required on success.** `200`/`204`/`206` = success; `501` (or absent) = server declined RDMA. Under `accelerated=true` the client treats a decline as a hard error (no automatic HTTP fallback today — see README). |
61+
| `x-amz-rdma-bytes-transferred` | response (GET) | Actual bytes moved over RDMA (may be `< requested` for ranged GETs). |
62+
| `ETag`, `x-amz-checksum-crc64nvme` | response (PUT) | Standard S3 result metadata. |
63+
64+
### Token format
65+
66+
```text
67+
x-amz-rdma-token: <descriptor>:<start_addr_hex>:<size_hex>
68+
```
69+
70+
- `<descriptor>` — the opaque, fixed-length RDMA descriptor minted by the
71+
client's RDMA provider (cuObject `cuMemObjGetRDMAToken`); it encodes the
72+
registered buffer's base address, max size, remote key and NIC routing
73+
(GID/LID/DCTN). The server splits it off (it is a known fixed length) and
74+
forwards exactly those bytes to its own RDMA provider (cuObjServer); it must
75+
not interpret them.
76+
- `<start_addr_hex>` — client buffer start address for this operation, 16-digit
77+
lowercase hex.
78+
- `<size_hex>` — transfer size in bytes, 16-digit lowercase hex.
79+
80+
Appending the per-operation start address and size to the descriptor mirrors
81+
cuObject's own **IO descriptor** layout (its reference server reads
82+
`<descriptor><rem_buf_start>…<rem_msg_size>…`); it is not an endpoint-specific
83+
addition. Because the HTTP body is empty (`Content-Length: 0`), the token is the
84+
only place these per-request fields can travel. The two trailing fields are
85+
`:`-separated and base-16; the server splits them off the end and treats the
86+
remaining prefix as the descriptor. A trailing `;` (some clients append one)
87+
must be tolerated.
88+
89+
## Request flow (server obligations)
90+
91+
1. **Authenticate** the request with standard SigV4. `x-amz-rdma-token` is part
92+
of the signed headers; `x-amz-content-sha256` is `UNSIGNED-PAYLOAD`.
93+
2. **Detect** an RDMA request by the presence of `x-amz-rdma-token`; parse it
94+
into `(descriptor, start_addr, size)`.
95+
3. **Decide** whether RDMA can be honored (fabric reachable, buffer registrable,
96+
object/permissions valid).
97+
- **If not** → respond `x-amz-rdma-reply: 501`. You MAY include a normal HTTP
98+
error body (do not force `Content-Length: 0`). The `501` signal is what lets
99+
a client safely decide what to do; today this client treats it as an error
100+
under `accelerated=true` (it does not auto-retry over HTTP — see README), but the
101+
signal is required so future clients can fall back transparently.
102+
4. **Transfer** using the cuObject server APIs (register a local buffer, then):
103+
- GET / RANGE_GET → `RDMA_WRITE` the requested bytes into the client buffer.
104+
- PUT / UploadPart → `RDMA_READ` the bytes from the client buffer and persist
105+
them as the object/part.
106+
5. **Respond** on success:
107+
- `x-amz-rdma-reply: 200` (full GET/PUT), `204` (PUT, no content), or `206`
108+
(ranged GET partial).
109+
- `x-amz-rdma-bytes-transferred: <n>` for GET.
110+
- `Content-Length: 0` and **no HTTP body** (the data went over RDMA — without
111+
this the client blocks waiting for body bytes that never arrive).
112+
- `ETag` (and `x-amz-checksum-crc64nvme` if requested) for PUT.
113+
114+
## Completion semantics
115+
116+
`x-amz-rdma-reply` is **not** a strict requirement on the PUT success path — it
117+
is an outcome marker the server is free to use:
118+
119+
- **PUT success:** the server completes the `RDMA_READ` and returns a standard
120+
HTTP `200` + `ETag` (the body is empty; data moved out-of-band). The client
121+
treats `200` + a non-empty `ETag` as success. AIStor does not set
122+
`x-amz-rdma-reply` on this path, and a client must not require it.
123+
- **GET success:** the server returns `x-amz-rdma-reply: 200/206` and
124+
`x-amz-rdma-bytes-transferred` (the authoritative moved-byte count); the HTTP
125+
body is empty.
126+
- **Decline:** `x-amz-rdma-reply: 501` (or its absence on a non-RDMA server)
127+
signals the request was not honored over RDMA.
128+
129+
The data path's integrity is covered by RoCEv2 iCRC and, when requested, the
130+
`x-amz-checksum-crc64nvme` header — not by the (empty) HTTP-body content hash.
131+
132+
## Multipart uploads
133+
134+
UploadPart uses the same body-less, token-carrying PUT with the standard
135+
`?uploadId=<id>&partNumber=<n>` query parameters (`1 ≤ n ≤ 10000`). The part's
136+
`ETag`/checksum are returned as usual; CompleteMultipartUpload is an ordinary
137+
HTTP call.
138+
139+
## Compliance checklist
140+
141+
- [ ] Parse `x-amz-rdma-token` as `<descriptor>:<addr16>:<size16>` (tolerate a
142+
trailing `;`).
143+
- [ ] Authenticate with SigV4 treating the body as `UNSIGNED-PAYLOAD`.
144+
- [ ] GET ⇒ `RDMA_WRITE` into the client buffer; PUT ⇒ `RDMA_READ` from it.
145+
- [ ] Always set `x-amz-rdma-reply` (`200`/`204`/`206`) on success, `501` to decline.
146+
- [ ] Set `x-amz-rdma-bytes-transferred` on GET; force `Content-Length: 0` with
147+
no body on every success.
148+
- [ ] Return `ETag`/checksum for PUT; honor `Range` for GET (`206`).
149+
- [ ] Fall through to normal HTTP semantics for non-RDMA requests.
150+
151+
A vendor that satisfies this checklist is supported by the NIXL object plugin's
152+
default S3 client automatically — no NIXL-side engine or plugin is required.

0 commit comments

Comments
 (0)