Add IBUC network adaptor support

[chencjcj]

test-ibuc.txt

[gemini-code-assist]

Summary of Changes

Hello @chencjcj, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request introduces a new InfiniBand Unreliable Connection (IBUC) network adaptor, enhancing the system's networking capabilities. It involves significant refactoring of existing InfiniBand-related code into a shared common header, streamlining the codebase and enabling easier integration of future InfiniBand adaptors. The changes also update the build system to support the new adaptor and modify the network selection logic to incorporate IBUC.

Highlights

• New IBUC Network Adaptor: Introduced support for a new InfiniBand Unreliable Connection (IBUC) network adaptor, enhancing the system's networking capabilities. This includes adding a USE_IBUC flag to the Makefile and integrating IBUC-specific structures and functions.
• InfiniBand Common Code Refactoring: Centralized common InfiniBand structures, constants, and function declarations into a new header file ib_common.h. This improves code organization and reusability across different InfiniBand adaptors like IBRC and the newly added IBUC.
• Makefile and Adaptor Selection Updates: The Makefile now includes conditional compilation for the USE_IBUC flag and a new print statement for its status. The network adaptor selection logic in adaptor.cc has been updated to prioritize IBUC when enabled and UCX is not present.
• IBRC Adaptor Modernization: The ibrc_adaptor.cc file has been significantly refactored by moving many static InfiniBand utility functions and global variable definitions to ib_common.h, making them shared and non-static for broader use by other adaptors.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request introduces a new network adaptor for InfiniBand Unreliable Connected (IBUC) transport. A significant part of this work involves refactoring common InfiniBand code from the existing IBRC adaptor into a shared header ib_common.h, which is a good improvement for maintainability.

However, the new IBUC adaptor implementation has a critical design flaw. It attempts to use RDMA operations (like RDMA_WRITE) with Unreliable Connected (UC) queue pairs. The InfiniBand UC transport type does not support RDMA; it is a message-based transport that relies on send/recv semantics. This fundamental misunderstanding makes the current IBUC implementation non-functional. The communication protocol for this adaptor needs to be redesigned to correctly use ibv_post_send and ibv_post_recv for data transfer.

I've also found a smaller issue with unreachable code in the ibuc_adaptor.cc file. Please see the detailed comments for specifics.

[gemini-code-assist]

The implementation of flagcxIbucMultiSend incorrectly uses RDMA opcodes (IBV_WR_RDMA_WRITE and IBV_WR_RDMA_WRITE_WITH_IMM). The Unreliable Connected (UC) transport type, which this adaptor is intended for, does not support RDMA operations. Data transfer over UC queue pairs must use a message-passing model with IBV_WR_SEND.

This implementation appears to be a copy of the Reliable Connected (RC) logic and will fail at runtime. The entire data transfer mechanism for the IBUC adaptor needs to be redesigned to use a send/recv paradigm instead of RDMA.

[gemini-code-assist]

The comment on this line is incorrect. Unreliable Connected (UC) mode does not support IBV_WR_RDMA_WRITE. This operation is only valid for Reliable Connected (RC) and Reliable Datagram (RD) queue pairs. Using this opcode with a UC QP will result in a runtime error.

This indicates a fundamental misunderstanding of the UC transport's capabilities and invalidates the current FIFO notification mechanism, which relies on RDMA.

[MC952-arch]

LGTM

[mikethegoblin]

LGTM