Skip to content

Add IMEX for 6.12 NVIDIA R570 driver - #204

Merged
yeazelm merged 6 commits into
bottlerocket-os:developfrom
yeazelm:imex
Jul 22, 2025
Merged

Add IMEX for 6.12 NVIDIA R570 driver#204
yeazelm merged 6 commits into
bottlerocket-os:developfrom
yeazelm:imex

Conversation

@yeazelm

@yeazelm yeazelm commented Jul 8, 2025

Copy link
Copy Markdown
Contributor

Description of changes:
This adds IMEX as a subpackage that provides support for GB200 devices which require the IMEX process to coordinate efficient communication between devices in ComputeDomains. The DRA device plugin will use these binaries when creating channels but GB200 hardware is required. This DRA DaemonSet provides its own configuration file so only the binaries are provided. We do not provide a systemd service file either because the intention is to let the DRA driver manage this.

Testing done:
Built an aws-k8s-1.33-nvidia with this kit and confirmed the binaries run.

bash-5.1# nvidia-imex --version
IMEX version is: 570.148.08
bash-5.1# nvidia-imex-ctl
Unable to open configuration file at /etc/nvidia-imex/config.cfg
Usage: nvidia-imex-ctl [-i <IP> <PORT>|-u <unix domain socket path>|-c <nvidia-imex config.cfg path>] <-n|-N|-q|-s|-d|-m> [-t <timeout>]
Where:
       -i <IP> <PORT>: Connect to the IMEX daemon's control service at the specified ip address and port.
       -u <unix domain socket path>: Connect to the IMEX daemon's control service at the specified unix domain socket path.
       -c <nvidia-imex config path>: Path to nvidia-imex configuration file.  Default value is /etc/nvidia-imex/config.cfg
NOTE:  If none of -i, -u, -c are provided, -c is assumed with default value.
       -n: Continuously monitor the entire IMEX domain status.  Requires config file (default or via -c)
       -N: Gets the full status of the entire IMEX domain.  Requires config file (default or via -c)
       -q: Query the status of the IMEX daemon once and return.
       -s: Subscribe to the status of the IMEX daemon.  Can be used with -t to terminate after a certain amount of time.
       -d: Dump performance metrics of the IMEX daemon once and return.
       -m: Subscribe to performance metrics of the IMEX daemon.  Can be used with -t to terminate after a certain amount of time.
       -t: Timeout (in milliseconds) for the -s command to terminate.  Unspecified means it will run until IMEX terminates.
       -r: Terminate status subscription when IMEX daemon returns READY status.
       -a: Only used with -q and -c.  Query status of all nodes configured, based on the config file in -c (or default.
       -j: Output of the -n/-N command will be in JSON, rather than formatted.
NOTE: only one of -i, -u, or -c can be specified.  If none are specified, -c is assumed with the default nvidia-imex config file location.
      also, only one command can be executed.

Terms of contribution:

By submitting this pull request, I agree that this contribution is dual-licensed under the terms of both the Apache License, version 2.0, and the MIT license.

@yeazelm
yeazelm requested review from arnaldo2792 and rpkelly and removed request for arnaldo2792 July 11, 2025 17:20
Comment thread packages/kmod-6.12-nvidia-r570/Cargo.toml
Comment thread packages/kmod-6.12-nvidia-r570/kmod-6.12-nvidia-r570.spec Outdated
Comment thread packages/kmod-6.12-nvidia-r570/kmod-6.12-nvidia-r570.spec Outdated
Comment thread packages/kmod-6.12-nvidia-r570/kmod-6.12-nvidia-r570.spec Outdated
Comment thread packages/kmod-6.12-nvidia-r570/kmod-6.12-nvidia-r570.spec
Comment thread packages/kmod-6.12-nvidia-r570/kmod-6.12-nvidia-r570.spec Outdated
Matthew Yeazel added 5 commits July 14, 2025 18:54
The variable for matching NVIDIA's upstream naming of architecture is
becoming more useful than just for Fabric Manager. This changes the
variable to a more generic name that expands beyond Fabric Manager.

Signed-off-by: Matthew Yeazel <yeazelm@amazon.com>
The variable for matching NVIDIA's upstream naming of architecture is
becoming more useful than just for Fabric Manager. This changes the
variable to a more generic name that expands beyond Fabric Manager.

Signed-off-by: Matthew Yeazel <yeazelm@amazon.com>
The variable for matching NVIDIA's upstream naming of architecture is
becoming more useful than just for Fabric Manager. This changes the
variable to a more generic name that expands beyond Fabric Manager.

Signed-off-by: Matthew Yeazel <yeazelm@amazon.com>
The variable for matching NVIDIA's upstream naming of architecture is
becoming more useful than just for Fabric Manager. This changes the
variable to a more generic name that expands beyond Fabric Manager.

Signed-off-by: Matthew Yeazel <yeazelm@amazon.com>
The variable for matching NVIDIA's upstream naming of architecture is
becoming more useful than just for Fabric Manager. This changes the
variable to a more generic name that expands beyond Fabric Manager.

Signed-off-by: Matthew Yeazel <yeazelm@amazon.com>
@yeazelm

yeazelm commented Jul 14, 2025

Copy link
Copy Markdown
Contributor Author

^ Adds in the rename for fm_arch to nvidia_arch in separate commits along with addressing comments from the PR.

Comment thread packages/kmod-6.12-nvidia-r570/kmod-6.12-nvidia-r570.spec Outdated
This adds IMEX as a subpackage that provides support for GB200 devices
which require the IMEX process to coordinate efficient communication
between devices in ComputeDomains.

Signed-off-by: Matthew Yeazel <yeazelm@amazon.com>
Comment thread packages/kmod-6.12-nvidia-r570/Cargo.toml
@yeazelm
yeazelm merged commit ae31993 into bottlerocket-os:develop Jul 22, 2025
2 checks passed
@yeazelm
yeazelm deleted the imex branch July 22, 2025 21:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants