Add NCCL H2 collective and P2P mismatch diagnostics

[lzy-edu]

PR Category

Train

PR Types

New Features

Dependency

This is a stacked PR based on the H1 diagnostics PR: #1255

PR Description

This PR extends the CPU-side NCCL probe with H2 diagnostics for inconsistent collective calls and unmatched or incompatible point-to-point communication.

The main changes include:

• Compare collective APIs across communicator ranks for the same communication round.
• Compare collective count, datatype, reduction operation, and root parameters.
• Distinguish api_mismatch, parameter_mismatch, and root_mismatch.
• Record P2P peer, group, group-operation index, and P2P-operation index metadata.
• Build directional P2P edges from communicator-local source and destination ranks.
• Resolve unique send/recv pairs before comparing their count and datatype.
• Detect p2p_missing_counterpart when an expected send or receive is not observed.
• Detect p2p_call_mismatch when a uniquely matched send/recv pair has incompatible parameters.
• Report ambiguous_p2p_match when multiple candidates cannot be attributed deterministically.
• Prefer a matching per-communicator P2P operation index when rank files are observed in different polling order.
• Preserve conservative confidence and clock assumptions for time-window-based P2P evidence.
• Prevent a confirmed signature mismatch from being duplicated as a downstream H1 missing-enter finding.
• Add unit tests and real two-worker NCCL mismatch scenarios.

The H2 diagnostics distinguish between:

• Different NCCL collective APIs used in the same communicator round.
• Different count, datatype, reduction operation, or root parameters.
• A send without a corresponding receive.
• A receive without a corresponding send.
• A uniquely matched send/receive pair with incompatible parameters.
• Multiple possible P2P candidates that cannot be assigned safely.

Usage

Rebuild the preload probe after updating to this PR:

python -m flagscale.runner.tracing.build

Enable H2 timeouts under experiment.runner.tracing:

experiment:
  runner:
    no_shared_fs: false

    heartbeat:
      enabled: true

    tracing:
      enabled: true
      probe_library: /absolute/path/to/libflagscale_nccl_probe.so
      collective_timeout_s: 60
      delayed_enter_threshold_s: 30
      p2p_timeout_s: 60
      p2p_match_window_s: 30
      failure_grace_period_s: 60
      scan_interval_s: 1

p2p_match_window_s must not exceed p2p_timeout_s, and failure_grace_period_s must cover the enabled H1/H2 timeout window.

Start the FlagScale training task with the existing launch command. The runner continues to manage the preload probe and node-zero analyzer automatically.

Results

Results remain under:

<log_dir>/tracing/<run_id>/
├── analyzer.log
├── analyzer.pid
├── findings.jsonl
├── training.exit_code
└── rank_<rank>_pid_<pid>.jsonl

The expected H2 behavior is:

• A normal collective or uniquely compatible send/recv pair does not generate an H2 finding.
• Different collective APIs generate collective_signature_mismatch with mismatch_type: api_mismatch.
• Different count, datatype, or reduction operation generates collective_signature_mismatch with mismatch_type: parameter_mismatch.
• Different Broadcast or Reduce roots generate collective_signature_mismatch with mismatch_type: root_mismatch.
• A send or receive without a counterpart generates p2p_missing_counterpart.
• A uniquely matched P2P pair with incompatible parameters generates p2p_call_mismatch.
• Multiple possible candidates generate ambiguous_p2p_match rather than a forced mismatch attribution.

For example:

{
  "hang_type": "collective_signature_mismatch",
  "run_id": "20260730_120000.000000",
  "comm_uid_hash": "0123456789abcdef",
  "comm_seq": 42,
  "detected_at_unix_ns": 1785400000000000000,
  "details": {
    "mismatch_type": "parameter_mismatch",
    "signature_by_comm_rank": {
      "0": {
        "api": "ncclAllReduce",
        "count": 1024,
        "datatype": 6,
        "op": 0,
        "root": -1
      },
      "1": {
        "api": "ncclAllReduce",
        "count": 2048,
        "datatype": 6,
        "op": 0,
        "root": -1
      }
    },
    "entered_comm_ranks": [0, 1],
    "expected_nranks": 2,
    "confidence": "confirmed"
  }
}

The finding confirms that the observed NCCL call contracts differ. It does not automatically identify which rank is responsible or prove that the application-level root cause is a tensor-shape or pipeline-scheduling bug.

[Darryl233]

If H1 and H2 exist in the same collective，we may miss the H1

[lzy-edu]

Thanks for pointing this out. You’re right: the unconditional continue after reporting a collective signature mismatch could skip the H1 missing-enter check when only part of the communicator had entered.

I changed the flow so that the round is completed immediately only when all expected ranks have entered. If the observed ranks already have an H2 mismatch but other ranks are still missing, the analyzer keeps checking the incomplete round and reports both collective_signature_mismatch and collective_missing_enter after the timeout.

When all ranks have entered, it still reports only the H2 mismatch, so no incorrect H1 finding is added.

I also added a 3-rank regression test covering two mismatched calls with one missing rank. The relevant test suite passed with 80 tests.

[Darryl233]

A mismatch could happen when rank0 send data to both rank1 and rank2？

[lzy-edu]

This case is already separated by the P2P storage key. Each call is grouped by (run_id, comm_uid_hash, src_rank, dst_rank), so a Send from rank 0 to rank 1 and a Send from rank 0 to rank 2 are stored in different candidate lists and cannot be matched with each other.
I added a 3-rank regression test covering rank 0 sending to both rank 1 and rank 2. Both Send/Recv pairs are matched independently, including the case where the local p2p_op_index values differ across the two endpoints. No mismatch or missing-counterpart finding is produced.

[Darryl233]

After the first timeout, the event is marked as resolved and deleted; the late Recv cannot restore the original Send and is later reported as a missing Send？

[lzy-edu]

After reporting a missing counterpart, the original P2P call was immediately marked as resolved and removed. If its counterpart was read later, the analyzer could no longer match it and might report a reverse missing-counterpart finding.
I changed the analyzer to retain a reported P2P call for one bounded grace period using the existing p2p_timeout_s. During this period, the original finding remains preserved, but a late counterpart can still match the retained call and clear the internal state without producing a reverse finding.
If no counterpart arrives before the grace period expires, the retained state is removed to avoid unbounded memory growth.
I also added tests covering both cases: a late counterpart being absorbed without a reverse report, and an unmatched reported call being cleaned after the grace period.

[Darryl233]

Every p2p event will call _p2p_candidates once, and _p2p_candidates will scan all history calls. This is an O(N^2) complexity, and may cause huge cpu overhead as the number of history calls increases.

[lzy-edu]

The previous implementation scanned the full P2P history in _p2p_candidates() for every new event, and resolved calls were only removed during the periodic scan.
I changed the probe to generate a per-peer and per-direction P2P operation index. The analyzer now uses this index for direct counterpart lookup, so the normal matching path is O(1) on average. Resolved, mismatched, ambiguous, and expired calls are also removed from all active indexes immediately.
For older traces or events without a reliable per-peer index, I kept the time-window fallback, but it now searches only the neighboring time buckets instead of the complete history.
I added regression coverage for a large batch of indexed P2P calls and verified that resolved history does not accumulate. I also rebuilt the probe and ran the real 3-rank multi-target scenario three times on A100.