fix(perf): reinstate aws-ofi-nccl EFA plugin for EKS

[ko3n1g]

Background

The NVIDIA PyTorch base image no longer ships the aws-ofi-nccl plugin (the
NCCL ⇆ libfabric bridge for AWS EFA) and now defaults NCCL_NET_PLUGIN=spcx
(HPCX Spectrum-X). The EFA/libfabric userspace is still present in the image
(/opt/amazon/efa), but without the aws-ofi-nccl libnccl-net.so, NCCL cannot
run over EFA on EKS. The base even leaves a dangling
/etc/ld.so.conf.d/aws-ofi-nccl.conf → /opt/amazon/ofi-nccl/lib.

This restores the plugin and selects it for AWS jobs.

What changed

• Image build rebuilds aws-ofi-nccl from source (public
  github.com/aws/aws-ofi-nccl, pinned v1.17.3) against the reinstalled NCCL and
  the image's libfabric, installing it into /opt/amazon/ofi-nccl — the prefix the
  leftover aws-ofi-nccl.conf already registers.
• Launcher pins NCCL_NET_PLUGIN to that plugin for AWS jobs, overriding the
  base image's spcx default. GCP is untouched (still selects gIB).

Details

• docker/common/install_aws_ofi_nccl.sh — clones + builds the plugin
  (--with-libfabric=/opt/amazon/efa --with-cuda=/usr/local/cuda --with-nccl=/usr --enable-platform-aws), installs to /opt/amazon/ofi-nccl, refreshes the
  ldconfig entry, ldconfig. Version pinned via --AWS_OFI_NCCL_VER.
• docker/Dockerfile.fw_base — new ARG AWS_OFI_NCCL_VERSION=v1.17.3; runs the
  script in fw_toolkit_builder right after install_nccl.sh, so the plugin links
  against the reinstalled NCCL.
• scripts/performance/utils/csp_plugins.py — EKSEnvPlugin now
  setdefaults NCCL_NET_PLUGIN=/opt/amazon/ofi-nccl/lib/libnccl-net.so (absolute
  path — the image also ships an HPCX libnccl-net.so, so a bare soname is
  ambiguous). A recipe/user override still wins. GKEEnvPlugin unchanged.
• tests/unit_tests/scripts/performance/test_csp_plugins.py — new coverage for
  both CSP plugins (previously untested).

Tested

• pytest tests/unit_tests/scripts/performance/test_csp_plugins.py — asserts the
  EKS plugin pins aws-ofi + EFA, respects a pre-set override, and no-ops off
  Kubeflow; the GKE plugin never sets NCCL_NET_PLUGIN.
• Image build is validated by CI (requires the FW build + registry).

[claude]

Light review — looks solid. The change is well-scoped and clearly documented: rebuild aws-ofi-nccl in the FW image and pin NCCL_NET_PLUGIN to it in EKSEnvPlugin so NCCL uses EFA instead of the base image's new spcx default. The setdefault usage correctly lets a recipe/user override win, and the new unit tests cover the pin, the override-respect path, the off-Kubeflow no-op, and that GKE never touches the aws-ofi plugin. Nice tests.

A few points to verify:

1. (Question) Installed library filename. AWS_OFI_NCCL_NET_PLUGIN = "/opt/amazon/ofi-nccl/lib/libnccl-net.so" and the docker install must agree on the exact filename make install drops. Recent aws-ofi-nccl releases have installed the plugin as libnccl-net-ofi.so (with a libnccl-net.so symlink) in some build configurations. If v1.17.3 installs it under a different name, this absolute path is a dangling reference and NCCL silently falls back to TCP — the exact failure this PR aims to prevent, but now hard to notice. Worth confirming the built artifact name in ${PREFIX}/lib matches, e.g. add a post-install test -e "${PREFIX}/lib/libnccl-net.so" guard in install_aws_ofi_nccl.sh so a rename fails the build loudly instead of at runtime.

2. (Minor) Shebang placement. #!/bin/bash sits after the license header (line 14), matching the other docker/common/install_*.sh scripts. A shebang after comments is inert but harmless since the script is invoked via bash /opt/.... No action needed — just noting it matches convention.

Suggested test cases:

• test_eks_plugin_pins_aws_ofi_and_enables_efa
• test_eks_plugin_respects_preset_net_plugin
• test_eks_plugin_is_noop_off_kubeflow
• test_gke_plugin_never_sets_net_plugin_and_selects_gib
• test_gke_plugin_is_noop_without_networks

No perf/recipe configs under scripts/performance/configs/ are touched — No perf tests impacted.