[Proposal] libfabric-CXI backend for UCCL-EP on HPE Slingshot

[PanJason]

[Proposal]: libfabric-CXI backend for UCCL-EP on HPE Slingshot

Hi UCCL team,

We're from the Apertus / Swiss AI training effort, working on large-scale open MoE training on Alps (using HPE's Slingshot-11 Cassini NIC). For Apertus 2 we're evaluating replacements for the default NCCL all-gather-style MoE dispatcher, and UCCL-EP looks like a strong fit because Slingshot does not provide IBGDA and we can only use some CPU-proxy design for all2all communication.

On Alps the network interface uses libfabric's cxi provider, not typical ibverbs. Before we start prototyping, we'd like to ask:

1. Does UCCL-EP currently have a libfabric transport path? I saw EFA, Broadcom, and CX7 mentioned and supported. Is EFA implemented via libfabric (efa provider), and if so, is the transport layer abstracted enough that adding a cxi provider would be mostly provider substitution?
2. Is any CXI / Slingshot work already underway or planned? If so, we'd be glad to test on our platform for debugging and help with performance benchmarking.
3. If not, would a libfabric/CXI backend be welcomed? Our intent is to keep the UCCL-EP API unchanged and map proxy operations to libfabric RMA, starting with low-latency mode. This might be useful because, as far as we know, many EU data centers actually use Slingshot because they were initially built for HPC, not exactly LLM training. If we can demonstrate performance gains, it would benefit many users.

What we'd contribute

Assuming the answer to (3) is yes, we'd plan to upstream:

• A libfabric/CXI transport backend (low-latency mode first, then normal/high-throughput).
• Validation on real Apertus MoE training on Alps Slingshot
• Benchmarks vs. all-gather performances.

What we plan to verify first currently

The main open questions on the CXI side are CUDA HMEM registration, MR-mode handling (FI_MR_ENDPOINT / FI_MR_PROV_KEY), the write-data / immediate-data path (FI_CXI_ENABLE_WRITEDATA, cq_data_size, FI_REMOTE_CQ_DATA), and ordering between payload writes and control notifications on CXI. We plan to work through these in standalone libfabric tests before touching UCCL — starting with fi_info -p cxi -v on the actual training container, then host-memory RMA, CUDA-memory RMA, a write-data test, and a payload+control ordering stress test. Happy to share the full plan if useful.

From my understanding of the paper, UCCL-EP uses immediate-data control messages to emulate ordering semantics for EP modes. Correct me if wrong. This means the write-with-immediate number perf matters a lot which we have to test. If performance is low, we need to think of design changes.

Why we think this can be useful for both sides

• For UCCL: real Slingshot/CXI deployment and validation on an open large-scale MoE workload.
• For Apertus: a reusable open-source EP layer instead of an internal one-off.
• For the broader community: other EU data centers that want to train or serve large MoE models on Slingshot NICs.

Finally, thanks for building UCCL-EP, which benefits many people not use ib nics

[PanJason]

At people related to this issue: @FFGGSSJJ @andresnowak @youliang-zhu @haodong98

[MaoZiming]

@PanJason Thanks for reaching out! We are definitely interested.

We don’t currently have a generic libfabric backend in UCCL-EP, and there isn't ongoing CXI / Slingshot work on our side. That said, we would definitely welcome contributions toward a CXI / Slingshot backend and would be happy to work with you on it!

Could I confirm with you whether CXI supports AMO writes, and whether RDMA writes and AMO writes are ordered with respect to each other in send order?

If both are supported, UCCL-EP may be able to bypass the write-with-immediate path. We mainly use write-with-immediate when the network does not provide such guarantees. We have not observed write-with-immediate to be the bottleneck in practice.

cc @YangZhou1997

[PanJason]

@MaoZiming Thanks for your feedback. On paper, CXI does support AMO, but whether our provider (HPE Slingshot 11 Cassini) supports it needs some experimentation. I am testing them now and will come back to you tmr.

[PanJason]

@MaoZiming I was doing a bit of investigation over the weekend. Here is what I found for your question:

Support for AMO writes

Could I confirm with you whether CXI supports AMO writes

TL;DR: CXI with the implementation on our slingshot NIC supports AMO writes. The following operations are supported for both host memory (local DRAM to remote DRAM) and Device memory (local HBM to remote HBM)

libfabric op	IB verbs equivalent	Host memory	Device memory
fi_write	IBV_WR_RDMA_WRITE	Works	Works
fi_read	IBV_WR_RDMA_READ	Works	Works
fi_atomic(..., FI_ATOMIC_WRITE)	atomic-style remote write / set	Works	Works
fi_atomic(..., FI_SUM)	atomic add / fetch-add without fetch result	Works	Works
fi_fetch_atomic(..., FI_ATOMIC_READ)	atomic read / fetch	Works	Works
fi_fetch_atomic(..., FI_SUM)	fetch-and-add	Works	Works
fi_writedata	IBV_WR_RDMA_WRITE_WITH_IMM	Works	Works

Support for ordering

and whether RDMA writes and AMO writes are ordered with respect to each other in send order?

TL;DR from my results, if AMO is using fi_atomic(..., FI_ATOMIC_WRITE) then no (at least not between DRAM, works on my HBM test but I am not sure if that usage pattern is what UCCL needs).[This is wrong because of a bug in my benchmark code, which changes the src mem before the previous fi_atomic is acked from cq on the sender side] if AMO is using fi_atomic(..., FI_ATOMIC_WRITE) or fi_atomic(..., FI_SUM) then yes, at least on the same endpoint / TX context / transmit queue. This result holds empirically on our slingshot nic, which is not guaranteed by the doc

Relevant interfaces

The two basic interfaces are:

fi_write(...);                    // one-sided RMA write
fi_atomic(..., FI_ATOMIC_WRITE);  // atomic write / set
fi_atomic(..., FI_SUM);           // atomic add

The question is whether this sender-side order:

fi_write(payload)
fi_atomic(control)

implies this receiver-side visibility:

observe control
=> payload is already visible

Host-Memory Workflow

Sender:

fi_write payload[N] to remote host buffer
fi_atomic control[N] to remote host control word

Receiver:

poll/read local host control word
when control says N is ready, immediately check payload[N]

Result:

Control operation	Ordering mode	Result
FI_ATOMIC_WRITE	plain submission order	All passed
FI_SUM counter	plain submission order	All passed
FI_ATOMIC_WRITE	wait for matched payload completion before control (CQ completion)	All passed.
FI_ATOMIC_WRITE	payload FI_DELIVERY_COMPLETE before control	All passed.

Device-memory payload + DRAM control word

Sender:

fi_write payload[N] to remote GPU HBM
fi_atomic control[N] to remote host DRAM control word

Receiver:

polls host DRAM control_word
when control_word >= seq:
      launches CUDA kernel
      CUDA kernel reads remote_payload[seq] in local GPU HBM
      cudaMemcpy check result back to host

Result:

Payload target	Control target	Control operation	Ordering mode	Result
GPU HBM	host DRAM	FI_ATOMIC_WRITE	plain submission order	Passed
GPU HBM	host DRAM	FI_SUM	plain submission order	Passed
GPU HBM	host DRAM	FI_ATOMIC_WRITE	matched payload completion before control (CQ completion)	Passed.
GPU HBM	host DRAM	FI_ATOMIC_WRITE	payload FI_DELIVERY_COMPLETE before control	Passed.

Device-memory payload + control word

Sender:

fi_write payload[N] to remote GPU HBM
fi_atomic control[N] to remote GPU HBM control word

Receiver variants:

# Host-observed variant
polls GPU HBM control_word via cudaMemcpy
when control_word is observed:
      copies/checks GPU HBM payload from host side

# GPU-observed variant
launches CUDA kernel before sender starts
CUDA kernel spins on GPU HBM control_word
when control_word is observed:
      same CUDA kernel immediately reads/checks GPU HBM payload
      cudaMemcpy check result back to host

Result:

Payload target	Control target	Control operation	Ordering mode	Receiver observation	Result
GPU HBM	GPU HBM	fi_write	plain submission order	host observes control via CUDA copy	Passed
GPU HBM	GPU HBM	fi_write	plain submission order	CUDA kernel observes control	Passed
GPU HBM	GPU HBM	FI_ATOMIC_WRITEcontrol	plain submission order	host observes control via CUDA copy	Passed
GPU HBM	GPU HBM	FI_ATOMIC_WRITEcontrol	plain submission order	CUDA kernel observes control	Passed

Note: this device-memory control test was mainly control-style, not the full per-sequence UCCL queue protocol. FI_SUM targeting GPU HBM was tested as an atomic capability, but not yet as the full payload-write + GPU-HBM-control ordering pattern.

Conclusion

Empirically yes on Alps CXI backed by slingshot. We did not see any write/AMO reordering with any of the configuration combination shown in appendix in any of the 5 runs each of which contains 1m iterations. When the AMO/control was observed, the payload write was visible on host or GPU.

Nevertheless, I would not make it as an unqualified libfabric generic assumption.

It would also be good if you can share with me some pseudo code about how UCCL can possibly utilize this write + AMO pattern so I can do some more accurate tests. I can check the code by myself as well.

Appendix:

Different Configurations I have tested for the empirical results:

Memory:

1. Host DRAM only
2. GPU HBM payload + Host DRAM control
3. GPU HBM only

Receiver Validation:

1. Host CPU checks host DRAM payload/control
2. Host CPU observes host DRAM control, then launches CUDA kernel to check GPU HBM payload
3. Host CPU observes GPU HBM control via CUDA copy, then checks GPU HBM payload
4. CUDA kernel spins on GPU HBM control, then checks GPU HBM payload in-kernel

MR Layout:

1. Split MR: payload and control registered separately
2. Shared MR: payload and control in one registered region

Operation Pattern:

1. 1 fi_write(payload) -> fi_atomic(FI_ATOMIC_WRITE control)
2. 1 fi_write(payload) -> fi_atomic(FI_SUM control)
3. N fi_write(payload) -> fi_write(control)
4. N fi_write(payload) -> fi_atomic(FI_ATOMIC_WRITE control)

API Spelling:

1. fi_write + fi_atomic
2. fi_writemsg + fi_atomicmsg
3. fi_write + fi_writedata

HMEM / Registration:

1. No HMEM flag for host memory
   2.FI_HMEM_SYSTEM for host memory
2. FI_HMEM_CUDA for GPU HBM

Ordering Request:

1. No requested ordering, tx_order=0
2. FI_ORDER_RMA_WAW (fi_write only)
3. FI_ORDER_WAW (more general for fi_write anf fi_atomic)

All the combo passed the tests on the same endpoint / TX context / transmit queue.

PS: By the doc, both FI_ORDER_RMA_WAW and FI_ORDER_WAW actually just preserves the order of operations over the same mem addresses not operations across different mem addresses but both work empirically, even without negotiation ini initialization.

Concurrency / Stress Shape:

• Single pair, 2 nodes, 1 rank per node
• Four concurrent cross-node GPU pairs
• Depth 1, 16, 256
• Payload sizes: 8 B, 64 B, 4 KiB, 1 MiB
• Iterations up to 1M for small payloads

[PanJason]

@MaoZiming I am working on porting UCCL now and I am checking the code. I have some questions:
Why do we have to use amo here to increment the tail here ? What are the issues if we replace this with a raw write of the tail to the remote? Because in my understanding, the rdma_channel_tail layout is (channel_id, src_rdma_rank) and each nvl_rank only writes to the corresponding nvl_rank, so only one lane in the sender warp of the cuda block with the specific channel_id will write to this remote?

[fergusfinn]

hi @PanJason. We're working on the Isambard cluster in the UK, from what i can tell it has a similar shape - libfabric/CXI/Slingshot etc. We've been working on an internal experimental UCCL-EP fork (for wideEP inference, mostly) - it's pretty AI-driven, at the moment, but it is up and serving well in vLLM and SGLang benchmarks. I put our working (messy) version up here doublewordai#1

Design-wise it matches what you measured: we skip write-with-immediate entirely and use fi_write plus libfabric atomic adds for the tail/control path, leaning on the same-TX-context write→AMO ordering you measured.

would be good to compare notes or to work from either base to get libfabric/CXI support upstreamed to UCCL.

[PanJason]

@fergusfinn Hi, we have made fi_write + fi_atomic work as well. The problem is non-ideal performance. On our configuration, 4 GPUs per node, each GPU has 1 200Gb/s Slingshot NIC, the UCCL benchmark only give 70% of the NIC bw:

The benchmark shape follows the DeepSeek-V3 pretraining-style configuration where possible.
The command we are using:

bench/test_internode.py \
  --num-tokens 4096 \
  --hidden 7168 \
  --num-topk 8 \
  --num-topk-groups <nodes capped at 4> \
  --num-experts 288 \
  --test-ll-compatibility

Benchmark configuration:

• Tokens per batch: 4096.
• Hidden size: 7168.
• Top-k experts per token: 8.
• Total experts: 288.
• Local ranks / GPUs per node: 4.
• NIC topology: 1 Slingshot/CXI NIC per GPU/rank, with 25 GB/s per NIC
  (200 Gb/s).
• Dispatch dtypes: FP8 dispatch and BF16 dispatch were benchmarked separately.
• Combine dtype: BF16.
• Top-k groups: 4 for the 4-node and 8-node runs; 2 for the 2-node run because only two nodes participate.

The table reports the bottleneck bandwidth and transmit latency across node-local benchmark summaries. Internode rows report RDMA bandwidth; the intranode row reports NVL bandwidth. These are fresh validation runs with proxy command tracing disabled (UCCL_PROXY_TRACE=0), not reused allocation data.

Type	FP8 Dispatch #EP	Bottleneck bandwidth & latency	BF16 Dispatch #EP	Bottleneck bandwidth & latency	Combine #EP	Bottleneck bandwidth & latency
Intranode	4	346.08 GB/s (NVL), 317.16 us	4	392.64 GB/s (NVL), 542.17 us	4	364.32 GB/s (NVL), 584.31 us
Internode	8	15.21 GB/s (RDMA), 3967 us	8	15.27 GB/s (RDMA), 7667 us	8	25.99 GB/s (RDMA), 4501 us
Internode	16	17.45 GB/s (RDMA), 6222 us	16	18.26 GB/s (RDMA), 11536 us	16	18.63 GB/s (RDMA), 11388 us
Internode	32	15.40 GB/s (RDMA), 7835 us	32	15.80 GB/s (RDMA), 14787 us	32	15.72 GB/s (RDMA), 14882 us

• Did you also see the same low performance on your cluster?

We also did some porting on fi_writedata but I got mysterious ENTRY_NOT_FOUND errors from slingshot, which indicates event buffer on the NIC is overflowing. Did you ever see this?

• Also, we did some porting with fi_write + fi_write but have some data corruption. This is why I was asking the question before .

[fergusfinn]

Current analogous numbers:

bench/test_internode.py \
  --num-tokens 4096 \
  --hidden 7168 \
  --num-topk 8 \
  --num-topk-groups <nodes capped at 4> \
  --num-experts 288 \
  --test-ll-compatibility  

Type	FP8 Dispatch #EP	Bottleneck bandwidth & latency	BF16 Dispatch #EP	Bottleneck bandwidth & latency	Combine #EP	Bottleneck bandwidth & latency
Intranode	4	347.82 GB/s (NVL), 315.58 us	4	391.53 GB/s (NVL), 543.69 us	4	360.37 GB/s (NVL), 590.71 us
Internode	8	41.59 GB/s (RDMA), 1451 us	8	43.19 GB/s (RDMA), 2708 us	8	43.63 GB/s (RDMA), 2681 us

Hey - cluster is pretty congested at the moment, so i couldn't do the larger benchmarks. The measured ceiling we get from fi_write is something like ~24.3GB/s/NIC. I think amplification over the line rate is basically because the benchmarking script here has the denominator as the kernel time. Which then inflates the bandwidth since transfers are asynchronous with regards to the kernels? when using wall time we get something more like 21GB/s/NIC for the EP8 config.

Re. perf we got quite a bit faster by posting the writes asynchronously and then queueing the AMOs until their writes completed.

We also did some porting on fi_writedata but I got mysterious ENTRY_NOT_FOUND errors from slingshot, which indicates event buffer on the NIC is overflowing. Did you ever see this?

Haven't seen this, we had some problems that i can't quite remember getting fi_writedata working (might have been this) so we dropped it. Will look into it again if i get a chance

Also, we did some porting with fi_write + fi_write but have some data corruption. This is why I was asking the question before

There's some arcane stuff in the CXI spec about ordering of fi_writes, other writes and amos with respect to each other, from what i can tell, in principle they're unordered, which could cause corruption. https://ofiwg.github.io/libfabric/main/man/fi_cxi.7.html. it could be that the empirics above are just an artifact of the small-ish payload sizes. In practise, we wait for CQ completions before posting AMOs, which i think gives ordering. we saw exactly this with immediately-posted AMOs, it was clean at small token counts, corrupted output at ≥1024 tokens. Gating the AMO on the write's CQ completion seemed to fix it. I think this could have something to do with FI_CXI_RDZV_THRESHOLD changing behaviour at different message sizes.

[PanJason]

Thanks a lot for the performance numbers. I also ran your code here doublewordai#4 but got very poor performance. Could you please let me know which code you used to generate the numbers above? The number I got is as below:

Branch/run	FP8 dispatch	BF16 dispatch	Combine
Our current uccl/ep	15.33 GB/s RDMA	15.34 GB/s RDMA	32.84 GB/s RDMA
PR #4	7.32 GB/s RDMA	6.84 GB/s RDMA	32.48 GB/s RDMA

I am now checking what is going wrong here. If you can share your commit, it would great to debug. The code we used is here https://github.com/swiss-ai/uccl/tree/slingshot-dev
One more thing I would like to ask is whether you have some raw performance numbers of fi_atomic It seems on our platform the fi_atomic is extremely slow. I tried another version of batching the AMOs like what you have done but I got almost the same performance. Since we are using almost the same hardware, it would be really helpful if we could do some AB testing over the network primitives as well to check the slingshot/CXI configuration correctness.

Haven't seen this, we had some problems that i can't quite remember getting fi_writedata working (might have been this) so we dropped it. Will look into it again if i get a chance

I can provide scripts if you need for testing fi_writedata

i think gives ordering. we saw exactly this with immediately-posted AMOs, it was clean at small token counts, corrupted output at ≥1024 tokens.
We actually chose to immediately post the AMO, and so far, I have not seen the corruption you said (I assume the test_internode.py has some correctness checking? At least we did not see corruption here). The reason why we did not use the ack-then-AMO approach is also performance. I tested this technique and got quite low performance as well, roughly similar to what the table shows. I start to suspect that there are some issues with the FI_CXI_* configuration on our platform. Is it possible for you to share a list of FI_CXI_* configurations you used on you platform?

[fergusfinn]

great - yes i will test out your branch!

The tuning variables we have for FI_CXI_* are:

export FI_CXI_DEFAULT_CQ_SIZE="131072"
export FI_CXI_DEFAULT_TX_SIZE="2048"
export FI_CXI_DISABLE_NON_INJECT_MSG_IDC="1"
export FI_HMEM_CUDA_USE_GDRCOPY="1"
export FI_CXI_DISABLE_HOST_REGISTER="1"
export FI_MR_CACHE_MONITOR="userfaultfd"
export FI_CXI_RDZV_PROTO="alt_read"
export FI_CXI_RDZV_THRESHOLD="0"
export FI_CXI_RDZV_GET_MIN="0"
export FI_CXI_RDZV_EAGER_SIZE="0"
export FI_CXI_RX_MATCH_MODE="hybrid"

https://docs.isambard.ac.uk/user-documentation/guides/nccl/ (from here, for reference).

I guess this could explain differences in perf numbers. Let me run w/ your branch and get back to you. (my numbers were from this branch doublewordai#1 - but it has the one you used at its tip basically, so it should have the same perf)

Happy to run any testing scripts for the raw writedata etc. versions.

[fergusfinn]

Branch	FP8 Dispatch (EP8)	BF16 Dispatch (EP8)	Combine (EP8)
doublewordai/uccl @ cxi-ep (373ce5d)	41.62 GB/s (RDMA), 1450 us	43.70 GB/s (RDMA), 2678 us	43.94 GB/s (RDMA), 2662 us
swiss-ai/uccl @ slingshot-dev (1da9b40)	40.18 GB/s (RDMA), 1502 us	42.78 GB/s (RDMA), 2736 us	42.20 GB/s (RDMA), 2772 us

I think it might indeed be cluster tuning - it looks like the two branches are within a hair of one another

[PanJason]

Ah thanks a lot for the prompt response and numbers, which narrows the investigation scope a lot.
fi_write/fi_writedata performance test code:
https://pastebin.com/RVK5QD5L
And the slurm scripts we used to test the performance:
https://pastebin.com/yUnVBESL
You can adapt from here to see whether fi_writedata works on your cluster.

I am also doing another AB test now with your FI_CXI configs and your code and see what the results will look like.

[PanJason]

I tried with your FI_CXI_* configuration and what I got is still quite low.

Run	FP8 dispatch	BF16 dispatch	Combine
PR1 373ce5d, node 0	8.27 GB/s RDMA	7.90 GB/s RDMA	32.60 GB/s RDMA
PR1 373ce5d, node 1	8.24 GB/s RDMA	8.05 GB/s RDMA	32.89 GB/s RDMA

One thing I found not working on our cluster is FI_THREAD_ENDPOINT and my run had to fall back to FI_THREAD_DOMAIN . I guess likely this is not where the perf diff lies.

So likely this is a slingshot configuration issue. Given that we have the same hardware, a very next step is to compare the raw performance numbers of fi_* primitives such as fi_write and fi_atomic to see where the differences are coming from.

[fergusfinn]

ok, so with API_KIND=writedata, i get -38 (ENOSYS, function not implemented) running your script - this is the error i found when i tried this the first time.

What version of libfabric do you have on your system? I think the reason for my lack of writedata is is that the build here is very old - its showing 1.22, whereas it looks like fi_writedata landed in libfabric a couple months ago (https://github.com/ofiwg/libfabric/blob/main/NEWS.md#v250-fri-march-20-2026).

I built 2.5.1 locally, and i get the same error you described with fi_writedata:

CQ error: err=5 (Input/output error) prov_errno=18 provider='ENTRY_NOT_FOUND' flags=0x204

Then, perf numbers for fi_write, for both libfabric versions, 'our env' is the one from above, 'default env' is the one that's defaulted in the scripts with our FI_CXI_* toggled off. Swept both slots and payload sizes because i got some weird data, but its confined to 1.22

Payload	Buffer	1.22 + our env	1.22 + default env	2.5.1 + our env	2.5.1 + default env
4 KiB	16 MiB	4.4	4.3	4.4	4.5
16 KiB	64 MiB	17.0	16.9	16.8	16.7
64 KiB	256 MiB	21.3	20.8	21.0	21.2
256 KiB	1 GiB	20.3	20.3	20.3	20.3
320 KiB	1.25 GiB	20.4	20.4	20.4	20.4
512 KiB	2 GiB	1.7	1.7	20.2	20.2
1 MiB	4 GiB	2.9	2.8	19.7	19.7

SLOTS	Buffer	1.22 + our env	1.22 + default env	2.5.1 + our env	2.5.1 + default env
4096	1 GiB	20.3	20.3	20.3	20.3
5120	1.25 GiB	20.2	20.2	20.2	20.1
6144	1.5 GiB	1.7	1.7	19.6	20.2
7168	1.75 GiB	1.7	1.2	20.1	20.1
8192	2 GiB	1.7	1.7	20.1	20.0

There's something up with large registered buffer sizes on 1.22, so at the least this has convinced me i should upgrade!

Then i reran in case libfabric version explained the perf difference on your cluster, it doesn't seem to:

Branch	libfabric	FP8 Dispatch	BF16 Dispatch	Combine
doublewordai/uccl @ cxi-ep	1.22 (system)	41.6 GB/s	43.7 GB/s	43.9 GB/s
doublewordai/uccl @ cxi-ep	2.5.1 (built)	41.5 GB/s	43.3 GB/s	44.0 GB/s
swiss-ai/uccl @ slingshot-dev	1.22 (system)	40.2 GB/s	42.8 GB/s	42.2 GB/s
swiss-ai/uccl @ slingshot-dev	2.5.1 (built)	38.6 GB/s	42.2 GB/s	41.6 GB/s