Wrong suggestions for OpenMPI and GCC(core)

[Flamefire]

Describe the bug

1. LMod suggest only GCCcore instead of GCC for loading OpenMPI together with a GCCcore module
2. It suggests incompatible combinations of GCCcore and GCC

To Reproduce

Reproducer for /tmp/modules: modules.tar.gz

$ ml CuPy OpenMPI
Lmod has detected the following error:  These module(s) or extension(s) exist but cannot be loaded as requested: "CuPy", "OpenMPI"
   Try: "module spider CuPy OpenMPI" to see how to load the module(s).
   Or load any one of these options:
      module load release/24.04 GCCcore/13.2.0 CuPy OpenMPI
      module load release/24.04 GCCcore/14.2.0 CuPy OpenMPI
      module load release/24.10 GCCcore/13.2.0 CuPy OpenMPI



$ module load release/24.04 GCCcore/13.2.0 CuPy OpenMPI
Lmod has detected the following error:  These module(s) or extension(s) exist but cannot be loaded as requested: "OpenMPI"
   Try: "module spider OpenMPI" to see how to load the module(s).
   Or load any one of these options:
      module load release/24.04 GCC/13.2.0 GCCcore/13.2.0 CuPy OpenMPI
      module load release/24.04 GCC/14.2.0 GCCcore/13.2.0 CuPy OpenMPI



$ module load release/24.04 GCC/14.2.0 GCCcore/13.2.0 CuPy OpenMPI
Lmod Warning: 
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
The following dependent module(s) are not currently loaded: GCCcore/14.2.0 (required by: GCC/14.2.0)
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------



Custom hook:
Module GCCcore/14.2.0 unloaded.
Modules GCC/14.2.0, GCCcore/13.2.0, CuPy/14, OpenMPI/5.6 and 1 dependency loaded.

Expected behavior

Only correct suggestions

Additional context

LMod 9.2.3 release version used.

Are the 3 empty lines at the end of each error expected?

Possibly related to #805 and #823

[mrcawood]

Hi @Flamefire, thanks again for the reproducer; I used your modules.tar.gz layout for the regression test.

I have a branch ready for you to try:

Branch: IS828-toolchain-suggest
https://github.com/TACC/Lmod/tree/IS828-toolchain-suggest

This branch is the current integration point for the load-failure suggestion work. It builds on the earlier fixes for related issues and adds the #828 fix on top:

Issue	What it addressed
#805	Disjoint toolchains (e.g. Python under GCCcore, QGIS under GCC/OpenMPI), suggest paths from the right hierarchy, not invalid cross-product commands
#823	Shorter suggestions when dependencies are already loaded; stack-root swap paths still listed when appropriate
#828	GCC / GCCcore toolchain handling, no GCCcore-only suggestions when GCC is required, and no mismatched GCC + GCCcore pairs in one command

#828 fix (your report)

When modules sit under different but related toolchain layers (CuPy → GCCcore, OpenMPI → GCC), Lmod now:

1. Treats same-version GCC and GCCcore as one toolchain when merging spider parent paths
2. Prefers GCC over GCCcore in suggested commands when both apply
3. Normalizes spider parent paths (drops self-referential tail entries that skewed path counts)
4. Omits an already-loaded / requested GCCcore from a suggestion when the path implies a different matching GCCcore via GCC
5. Validates multi-module suggestions against all failing modules, not only the one with the fewest paths

So for your scenarios:

ml CuPy OpenMPI
# → suggests release/24.04 GCC/13.2.0 … / GCC/14.2.0 … (not GCCcore-only)

ml release/24.04 GCCcore/13.2.0 CuPy OpenMPI
# → suggests GCC/13.2.0 … / GCC/14.2.0 … (not GCC/14.2.0 GCCcore/13.2.0)

Tests

New RT test using your module tree:

• rt/load_suggest_828, your three scenarios (including a positive load of release/24.04 GCC/14.2.0 CuPy OpenMPI/5.6)

Regression coverage on the same branch:

• rt/load_suggest_cmd_805, Followup on "Show command to load module on failure" #804 #805 disjoint hierarchies
• rt/load_suggest_823, Overcomplicated (wrong?) suggestions for loading failed module #823 minimal / preloaded suggestions (uses the same reproducer family as your tarball)

All three pass on my side.

Please try

If you can, check out IS828-toolchain-suggest and re-run:

ml CuPy OpenMPI
ml release/24.04 GCCcore/13.2.0 CuPy OpenMPI
ml release/24.04 GCC/14.2.0 CuPy OpenMPI   # or your preferred valid combo

Confirm that:

1. Suggestions use GCC/… where OpenMPI needs it (not GCCcore-only)
2. No suggested command mixes GCC/x.y with a non-matching GCCcore/a.b
3. The suggested commands actually load without the dependency warning you saw before

Once you confirm, I’ll merge to main and cut a new point release.

[Flamefire]

I'm a bit worried about the hard-coded GCC/GCCcore there. I mean the suggestion itself was impossible to use, so somewhere in LMod there must be information that the modules cannot be loaded together.
I fear that there are similar cases given the >= 3 toolchains (GCC, Intel, Clang) and deeper toolchains (GCC-OpenMPI, GCC-IntelMPI, ...) so if this case can be handled generically we won't see the same there.
Or is the information not there or to hard to collect?

[mrcawood]

You're right to push back on the hardcoding; that was the wrong approach.

The information is there. Your modulefiles already declare the relationships (GCC depends_on GCCcore, CuPy depends_on GCCcore, etc.). The suggestion code walks spider parent paths but never had access to depends_on metadata in the spider cache, so I took a shortcut with literal GCC/GCCcore names. That doesn't generalize to Intel, Clang, or deeper stacks like GCC-OpenMPI vs GCC-IntelMPI.

I've reworked IS828-toolchain-suggest to use the generic path:

• During the spider walk, depends_on is now recorded into the cache (it was effectively discarded before).
• When merging parent paths from multiple modules, compatibility is decided from those dependencies — e.g. {release, GCC/13.2.0} and {release, GCCcore/13.2.0} merge because GCC/13.2.0 depends on GCCcore/13.2.0, with no compiler names in the logic.
• When paths diverge at the same position, we prefer the element that depends_on the other (outer toolchain over inner prerequisite).
• If a path element has a fixed depends_on for module X but the suggestion would use a different version of X, that suffix gets dropped — same idea as the mixed GCC/GCCcore case, but driven from metadata.

The same RT tests still pass (load_suggest_828 with your module layout, plus the #805 and #823 regressions). I've pushed the updated branch, pull latest IS828-toolchain-suggest before re-testing.

Your three scenarios are still what I'd like you to check:

ml CuPy OpenMPI
ml release/24.04 GCCcore/13.2.0 CuPy OpenMPI
ml release/24.04 GCC/14.2.0 CuPy OpenMPI

Mainly: suggestions should use GCC/… where OpenMPI needs it, no mixed GCC/x + wrong GCCcore/y in one command, and the suggested loads should actually work without the dependency warning.

[Flamefire]

Your three scenarios are still what I'd like you to check:

IIUC your change includes a new "format" of the spider cache saved on disk? That would mean I cannot test that without "damaging" the spider cache used by the production LMod version.

What would be the easiest way to temporarily move the spider cache? Or does LMOD_IGNORE_CACHE ensure that the files won't be written to and the feature can still be tested without a "physical" cache?

[mrcawood]

Yes, the spider cache on disk picks up extra depT metadata (depends_on recorded during the spider walk). That’s additive, but a test run with the branch build can still overwrite your normal cache files under the default location.

LMOD_IGNORE_CACHE / module --ignore_cache is not enough. It skips reading the existing cache and rebuilds from modulefiles, but Lmod can still write cache files afterward. So production cache isn’t protected by ignore-cache alone.

Easiest safe test: point cache at a throwaway directory for that shell session:

export LMOD_CACHE_DIR=/tmp/lmod-cache-is828-test

then use the IS828-toolchain-suggest build as usual

ml CuPy OpenMPI
ml release/24.04 GCCcore/13.2.0 CuPy OpenMPI
ml release/24.04 GCC/14.2.0 CuPy OpenMPI

Nothing touches your real cache. Drop the env var (or open a new shell) when you’re done.

Alternative: temporarily rename the default cache dir (often ~/.cache/lmod), test, then rename it back. LMOD_CACHE_DIR is simpler.

You can combine --ignore_cache with a separate LMOD_CACHE_DIR if you want a cold rebuild into an isolated dir; the important part is not writing into the production cache path.

[Flamefire]

I got around to test the changes on our module tree:

$  module load NCCL OpenMPI CUDA       
Lmod has detected the following error:  These module(s) or extension(s) exist but cannot be loaded as requested: "NCCL", "OpenMPI"
   Try: "module spider NCCL OpenMPI" to see how to load the module(s).
   Or load any one of these options:
      module load GCC/13.3.0 NCCL OpenMPI CUDA
      module load GCC/14.2.0 NCCL OpenMPI CUDA
      module load release/25.06 GCC/13.3.0 NCCL OpenMPI CUDA
      module load release/25.06 GCC/14.2.0 NCCL OpenMPI CUDA
      module load release/24.04 NVHPC/24.9-CUDA-12.6.0 NCCL OpenMPI
      ... and 11 more options

$       module load GCC/13.3.0 NCCL OpenMPI CUDA
Lmod Warning: 
--------------------------------------------------------------------------------------------------------------------------
The following dependent module(s) are not currently loaded: CUDA/12.6.0 (required by: UCX-CUDA/1.16.0-CUDA-12.6.0, NCCL/2.22.3-CUDA-12.6.0)

So it works but runs into issues due to loading the highest version by default:

• NCCL/2.22.3-CUDA-12.6.0 depends on CUDA/12.6
• CUDA modules 12.6 and 13.0 are installed

--> Loading NCCL before CUDA might switch CUDA.

With LMOD_DISABLE_SAME_NAME_AUTOSWAP enabled this will be a hard error. But not sure if/how this should be handled given no explicit versions was given.
But it feels like a bug that you cannot C&P the given command.

Minor: "Or load any one of these options:" and "... and 15 more options" are not translated to German at least (above output doesn't show this due to LANG=C)
Maybe "use" or "run" might be better than "load" given that it shows full commands now

[mrcawood]

Thanks for testing on your tree, the NCCL/CUDA case is exactly the gap you described.
What went wrong: for module load NCCL OpenMPI CUDA, the suggestion suffix kept the user's unversioned short names (NCCL, CUDA). Copy-pasting module load GCC/13.3.0 NCCL OpenMPI CUDA then resolves CUDA to the default/highest install (13.0), while NCCL/2.22.3-CUDA-12.6.0 requires CUDA/12.6.0, hence the dependency warning (or a hard error with LMOD_DISABLE_SAME_NAME_AUTOSWAP).

And agreed: a suggested command should be copy-pasteable.

Fix (pushed to IS828-toolchain-suggest, commit 8aeed727): when building the suffix list for a suggestion, if the user requested an unversioned module and spider metadata has a fixed depends_on, and that dependency's short name is also in the same load request, we now:

1. Resolve the suffix to the versioned fullName from the spider cache for that toolchain path (not the bare short name).
2. Emit any such fixed dependencies before the dependent module (e.g. CUDA/12.6.0 before NCCL/2.22.3-CUDA-12.6.0).
   So the same request becomes something like:

module load GCC/13.3.0 CUDA/12.6.0 NCCL/2.22.3-CUDA-12.6.0 OpenMPI

instead of … NCCL OpenMPI CUDA. That command should load without the CUDA mismatch.
RT coverage, extended rt/load_suggest_828 with a minimal fixture (CUDA 12.6 + 13.0, NCCL depends_on("CUDA/12.6.0")):

Step	Request	Assert
9	load NCCL OpenMPI CUDA	Suggestion pins CUDA/12.6.0 and NCCL/2.22.3-CUDA-12.6.0, not bare NCCL/CUDA
11, 13	Copy-paste the pinned suggestion	Clean load, no dependency warning
15	load NCCL CUDA (no OpenMPI)	Same pinning for the NCCL/CUDA pair
17, 19	Copy-paste that suggestion	Clean load

Please pull latest IS828-toolchain-suggest and re-test with an isolated cache:

export LMOD_CACHE_DIR=/tmp/lmod-cache-is828-$$
module load NCCL OpenMPI CUDA
# copy-paste the suggested command

On the i18n note ("Or load any one of these options:" / "... and N more options"), noted; separate from this suffix fix.

[Flamefire]

Unfortunately that didn't work. The output is unchanged comparing to the previous commit:

   Or load any one of these options:
      module load GCC/13.3.0 NCCL OpenMPI CUDA
      module load release/25.06 GCC/13.3.0 NCCL OpenMPI CUDA
      module load release/24.04 NVHPC/24.9-CUDA-12.6.0 NCCL OpenMPI
      module load release/2026 GCC/14.3.0 NCCL OpenMPI CUDA
      module load release/24.04 GCC/12.3.0 NCCL OpenMPI CUDA

As I've seen that in the -D output I ran those which might be related or a separate issue:

$ ml spider '^NCCL$'
Lmod has detected the following error:  Unable to find: "^NCCL$".
$ ml spider '^NCCL'
Lmod has detected the following error:  Unable to find: "^NCCL".
$ ml spider 'N.*CL'
Lmod has detected the following error:  Unable to find: "N.*CL".
$ ml spider 'NCCL'

-----------------------------------------------------------------------------------------------------------------------
  NCCL:
-----------------------------------------------------------------------------------------------------------------------
    Description:
      The NVIDIA Collective Communications Library (NCCL) implements multi-GPU and multi-node collective communication primitives that are performance optimized for NVIDIA GPUs.

     Versions:
        NCCL/2.12.12-CUDA-11.7.0
        NCCL/2.18.3-CUDA-12.1.1
        NCCL/2.18.3-CUDA-12.4.0
        NCCL/2.20.5-CUDA-12.4.0
        NCCL/2.22.3-CUDA-12.6.0
        NCCL/2.26.2-CUDA-12.6.0
        NCCL/2.26.6-CUDA-12.6.0
        NCCL/2.26.6-CUDA-12.8.0
        NCCL/2.27.5-CUDA-12.6.0
        NCCL/2.27.7-CUDA-12.9.1

BTW: As the comment sounds like AI, which one are you using if any? Just curious.

[mrcawood]

Thanks for re-testing. You were right that 8aeed72 did not change the NCCL/CUDA case on your tree. The pin path was too narrow, so the suffix still kept the bare short names.

That is fixed on the tip of IS828-toolchain-suggest (e1a57f2). Unversioned names in a suggestion are now rewritten to the spider fullName for that toolchain path. Fixed depends_on modules that are also in the same request are emitted before the dependent (so CUDA comes before NCCL). If several versions match the same path, the highest wV wins, and the matching CUDA version is pinned with it.

For your example, the tip should look like:

module load GCC/13.3.0 CUDA/12.6.0 NCCL/2.22.3-CUDA-12.6.0 OpenMPI/...

instead of:

module load GCC/13.3.0 NCCL OpenMPI CUDA

I have added (many) cases to the regression test; the coverage is very strong now, so I hope we have caught all the edges. Please re-test with an isolated cache again.

The module spider '^NCCL$' / regex cases you saw in -D are separate: spider name lookup does not treat those patterns as regexes unless you use -r. Happy to track that in another issue if you want.

PS: I use the Cursor IDE with the included Composer model with the Pro subscription. They just doubled the included tokens, which is a very good value right now.

[Flamefire]

The module spider '^NCCL$' / regex cases you saw in -D are separate: spider name lookup does not treat those patterns as regexes unless you use -r. Happy to track that in another issue if you want.

Is that then an issue inside LMods logic? Those above are done manually but IIRC -D revealed such regex searches with -r which I thought might be why it failed to find the right module

[mrcawood]

Good catch on the -D output. That is Lmod’s own logic, not a stray search.

On a failed load, before it prints suggestions, Lmod classifies each requested name as known or unknown by running an internal spider probe with --regexp and an anchored pattern, e.g. spider '^NCCL$'. That is why -D shows those regex spider calls. They are existence checks for the suggestion path; they are not how the versioned suffix is chosen.

So your manual:

ml spider '^NCCL$'

fails because that is a literal name without -r. With regexp it should work:

ml -r spider '^NCCL$'

That probe finding NCCL is expected on your tree (your suggestions already listed NCCL/OpenMPI paths). It was not why 8aeed727 left bare NCCL/CUDA in the suffix. That was the narrow pin path, fixed in e1a57f216.

If you have not already, a quick re-check of:

export LMOD_CACHE_DIR=/tmp/lmod-cache-is828-test
ml NCCL OpenMPI CUDA

on tip IS828-toolchain-suggest would be the useful next data point. Looking for versioned CUDA/… and NCCL/… in the suggested command, and a clean copy-paste load.

[Flamefire]

Still the issue with wrong NCCL-CUDA-Combo

$ ml NCCL OpenMPI CUDA
Lmod has detected the following error:  These module(s) or extension(s) exist but cannot be loaded as requested: "NCCL", "OpenMPI"
   Try: "module spider NCCL OpenMPI" to see how to load the module(s).
   Or load any one of these options:
      module load GCC/13.3.0 NCCL OpenMPI/5.0.3 CUDA/13.0.0
      module load release/25.06 GCC/13.3.0 NCCL OpenMPI/5.0.3 CUDA/13.0.0
      module load release/24.04 NVHPC/24.9-CUDA-12.6.0 NCCL OpenMPI/5.0.3
      module load release/2026 GCC/14.3.0 NCCL OpenMPI/5.0.8 CUDA/12.9.1
      module load release/24.04 GCC/12.3.0 NCCL OpenMPI/4.1.5 CUDA/12.6.0
      ... and 2 more options
$ module load GCC/13.3.0 NCCL OpenMPI/5.0.3 CUDA/13.0.0
Lmod Warning: 
---------------------------------------------------------------------------------------------------------------------------------
The following dependent module(s) are not currently loaded: CUDA/12.6.0 (required by: UCX-CUDA/1.16.0-CUDA-12.6.0, NCCL/2.22.3-CUDA-12.6.0)
---------------------------------------------------------------------------------------------------------------------------------

Module CUDA/12.6.0 unloaded.
Modules GCC/13.3.0, NCCL/2.22.3-CUDA-12.6.0, OpenMPI/5.0.3, CUDA/13.0.0 and 18 dependencies loaded.

So looks as if the last change didn't do anything but verified e1a57f2 is included

We have the following NCCL modules:

/software/modules/genoa/r2026/all/Compiler/GCCcore/14.3.0/NCCL/:
2.27.7-CUDA-12.9.1.lua

/software/modules/genoa/r24.04/all/Compiler/GCCcore/12.3.0/NCCL/:
2.18.3-CUDA-12.1.1.lua

/software/modules/genoa/r24.04/all/Compiler/GCCcore/13.2.0/NCCL/:
2.18.3-CUDA-12.4.0.lua

/software/modules/genoa/r24.04/all/Compiler/GCCcore/13.3.0/NCCL/:
2.22.3-CUDA-12.6.0.lua

/software/modules/genoa/r24.10/all/Compiler/GCCcore/13.2.0/NCCL/:
2.20.5-CUDA-12.4.0.lua

/software/modules/genoa/r25.06/all/Compiler/GCCcore/13.3.0/NCCL/:
2.22.3-CUDA-12.6.0.lua

/software/modules/genoa/r25.06/all/Compiler/GCCcore/14.2.0/NCCL/:
2.26.6-CUDA-12.8.0.lua

What I noticed: Even with export LMOD_CACHE_DIR=/tmp/lmod-cache-is828-test it used ~/.cache/lmod which I only noticed when it suggested NCCL modules I only had in my test build trees that didn't exist anymore