Skip to content

bnxt_re/lib: Support direct verbs#1654

Open
sbasavapatna wants to merge 5 commits intolinux-rdma:masterfrom
sbasavapatna:dv-upstream
Open

bnxt_re/lib: Support direct verbs#1654
sbasavapatna wants to merge 5 commits intolinux-rdma:masterfrom
sbasavapatna:dv-upstream

Conversation

@sbasavapatna
Copy link
Copy Markdown

@sbasavapatna sbasavapatna commented Nov 3, 2025 •
edited
Loading

Hi,

This patchset supports Direct Verbs in the bnxt_re library.

This is required by vendor specific applications that need to manage
the HW resources directly and to implement the datapath in the
application.

To support this, the library and the driver are being enhanced to
provide Direct Verbs using which the application can allocate and
manage the HW resources (Queues, Doorbell etc) . The Direct Verbs
enable the application to implement the control path.

Patch#1 Update kernel headers
Patch#2 Export HSI file
Patch#3 Support DBR and UMEM verbs
Patch#4 Support CQ verbs
Patch#5 Add man pages

Thanks,
-Harsha

@jgunthorpe
Copy link
Copy Markdown
Member

jgunthorpe commented Dec 17, 2025

Please take the debug_file stuff out of here, I'm not keen on every driver implementing their own stuff. We already have tracing frameworks supported, why not use one?

jgunthorpe
jgunthorpe reviewed Dec 17, 2025
View reviewed changes
providers/bnxt_re/bnxt_re_dv.h Outdated
struct bnxt_re_dv_db_region_attr *attr);
int bnxt_re_dv_get_default_db_region(struct ibv_context *ibvctx,
struct bnxt_re_dv_db_region_attr *out);
void *bnxt_re_dv_umem_reg(struct ibv_context *ibvctx,
Copy link
Copy Markdown
Member

@jgunthorpe jgunthorpe Dec 17, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

void *? That shouldn't be. Why do you have these functions when there is no kernel side?

@sbasavapatna
Copy link
Copy Markdown
Author

sbasavapatna commented Dec 24, 2025 via email

@sbasavapatna
Copy link
Copy Markdown
Author

sbasavapatna commented Dec 24, 2025 via email

@jgunthorpe
Copy link
Copy Markdown
Member

jgunthorpe commented Jan 7, 2026

So give it an opaque type, don't just randomly use void *s.

Again,why is this stuff here when there is no kernel side for umems?

@sbasavapatna
Copy link
Copy Markdown
Author

sbasavapatna commented Jan 8, 2026 via email

@sbasavapatna
Copy link
Copy Markdown
Author

sbasavapatna commented Jan 8, 2026

On Thu, Jan 8, 2026 at 1:28 AM Jason Gunthorpe @.> wrote: jgunthorpe left a comment (linux-rdma/rdma-core#1654) <#1654 (comment)> So give it an opaque type, don't just randomly use void s.
Are you suggesting to use a pointer to the forward declaration of that structure? Is that what you meant by "give it an opaque type"? Please clarify.
Again,why is this stuff here when there is no kernel side for umems?
The application registers the addr and the total size of the memory area with the library, initially using this API. Later during each object creation, the library uses this information to provide the actual va (base-addr + offset) for mapping/pinning, to the kernel driver. This has been captured in the kernel driver review thread: @./
…
— Reply to this email directly, view it on GitHub <#1654 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AGTY2BIR6EFHFZXC2CENTI34FVQP5AVCNFSM6AAAAACK7AEXQ2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTOMRQGUZTOMBUGI . You are receiving this because you authored the thread.Message ID: @.**>

Here's the pointer to the kernel review thread for reference:
https://lore.kernel.org/linux-rdma/CAHHeUGXGTfsK66DOGdE6Y5VYVaOR=1YA6b2inds6_EkPSwTBuA@mail.gmail.com/

@jgunthorpe
Copy link
Copy Markdown
Member

jgunthorpe commented Jan 9, 2026

Just have some 'struct bnxt_umem' as a forward declaration that never sees an actual definition. It is now an opaque type you can use instead of void.

You also need to write man pages for these functions, see providers/mlx5/man/mlx5dv_devx_umem_reg.3.md for instance.

@sbasavapatna
Copy link
Copy Markdown
Author

sbasavapatna commented Jan 13, 2026 via email

@sbasavapatna
Copy link
Copy Markdown
Author

sbasavapatna commented Jan 15, 2026

While looking into changes needed in the bnxt_re kernel driver to process DBR passed through driver specific attributes (during QP creation), I noticed that the corresponding libibverbs support seems to be missing in cmd_qp.c.
That is, ibv_cmd_create_qp_ex() doesn't support "ibv_command_buffer *driver" arg that should be linked to the main command buffer that it creates. This is supported in CQ creation command though (ibv_cmd_create_cq_ex2()).

@jgunthorpe
Copy link
Copy Markdown
Member

jgunthorpe commented Jan 15, 2026

As the first user you'd need to do a similar thing as CQ.

@sbasavapatna
Copy link
Copy Markdown
Author

sbasavapatna commented Jan 17, 2026 via email

@sbasavapatna sbasavapatna force-pushed the dv-upstream branch 2 times, most recently from 294a076 to 3d8c351 Compare February 3, 2026 05:11
@sbasavapatna sbasavapatna force-pushed the dv-upstream branch 2 times, most recently from d43117e to 362804c Compare February 11, 2026 12:59
@sbasavapatna sbasavapatna reopened this Apr 7, 2026
@sbasavapatna sbasavapatna force-pushed the dv-upstream branch 2 times, most recently from 51826f6 to 7a10dd5 Compare April 7, 2026 08:58
sbasavapatna and others added 3 commits April 8, 2026 13:53
@sbasavapatna
Update kernel headers
3a93898 
To commit: eb15cffa1520 ("RDMA/bnxt_re: Support application specific
CQs").

Signed-off-by: Sriharsha Basavapatna <sriharsha.basavapatna@broadcom.com>
Reviewed-by: Selvin Xavier <selvin.xavier@broadcom.com>
@sbasavapatna
bnxt_re/lib: Direct Verbs: Export HSI file
8073f05 
Extract HSI definitions (WQE/CQE structures and enums) from
bnxt_re-abi.h into a standalone bnxt_re-hsi.h and add it to
publish_headers() so it is installed under infiniband/.

This will make HSI definitions available to DV applications,
that implement their own datapath.

Signed-off-by: Sriharsha Basavapatna <sriharsha.basavapatna@broadcom.com>
Reviewed-by: Selvin Xavier <selvin.xavier@broadcom.com>
@kaleshap86 @sbasavapatna
bnxt_re/lib: Direct Verbs: Support DBR and UMEM verbs
ae54649 
The following Direct Verb (DV) APIs have been implemented in
this patch.

Doorbell Region Direct Verbs:

- bnxt_re_dv_alloc_db_region():
  This will allow the appliation to create extra doorbell regions
  and use the associated doorbell page index in dv_create_qp() and
  use the associated DB address while ringing the doorbell.

- bnxt_re_dv_free_db_region():
  Free the allocated doorbell region.

- bnxt_re_dv_get_default_db_region():
  Return the default doorbell page index and doorbell page address
  associated with the ucontext.

Umem Registration Direct Verbs:

- bnxt_re_dv_umem_reg():
  Register the user memory to be used by the application with
  the library. Application can register a large chunk of memory and
  use it during subsequent resource creation DV APIs.

  Note that the API terminates in the library and the app specified
  memory params (addr, len) are saved and a umem-handle is returned.
  That is, there is no ioctl to the driver at this point to map/pin
  the user memory.

  This memory is mapped/pinned later when the application creates
  the required resources (CQ/QP) using respective direct verbs.
  This is implemented in the next patch in this series.

- bnxt_re_dv_umem_dereg():
  Deregister the user memory specified by the umem-handle.

Co-developed-by: Sriharsha Basavapatna <sriharsha.basavapatna@broadcom.com>
Signed-off-by: Sriharsha Basavapatna <sriharsha.basavapatna@broadcom.com>
Signed-off-by: Kalesh AP <kalesh-anakkur.purayil@broadcom.com>
Reviewed-by: Selvin Xavier <selvin.xavier@broadcom.com>
@sbasavapatna
bnxt_re/lib: Direct Verbs: Support CQ verbs
4f04159 
The following Direct Verb (DV) APIs have been implemented in
this patch. These are implemented by enhancing the driver
specific udata in existing verbs.

CQ Direct Verbs:

- bnxt_re_dv_create_cq():
  Create a CQ of requested size (cqe). The application must have
  already registered this memory using bnxt_re_dv_umem_reg().
  The CQ umem-handle and umem-offset provided by the application
  are translated into an address for mapping and passed to the
  driver.

- bnxt_re_dv_destroy_cq():
  Destroy the DV_CQ created earlier.

Note:
Some applications might want to allocate memory for all CQs in one big
chunk and then register that entire memory once using dv_umem_reg().
At the time of creating each individual resource, the application
should pass a specific offset/length in the umem registered memory.

Signed-off-by: Sriharsha Basavapatna <sriharsha.basavapatna@broadcom.com>
Co-developed-by: Kalesh AP <kalesh-anakkur.purayil@broadcom.com>
Signed-off-by: Kalesh AP <kalesh-anakkur.purayil@broadcom.com>
Reviewed-by: Selvin Xavier <selvin.xavier@broadcom.com>
@sbasavapatna
bnxt_re/lib: Direct Verbs: Add man pages
24a1e0f 
Document direct verbs (DV) feature and related DV APIs .

Signed-off-by: Sriharsha Basavapatna <sriharsha.basavapatna@broadcom.com>
Reviewed-by: Selvin Xavier <selvin.xavier@broadcom.com>
@sbasavapatna sbasavapatna requested a review from jgunthorpe April 8, 2026 08:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@jgunthorpe jgunthorpe Awaiting requested review from jgunthorpe

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@sbasavapatna @jgunthorpe @kaleshap86