Add CPU-side NCCL H1 missing-enter diagnostics

[lzy-edu]

PR Category

Train

PR Types

New Features

Dependency

This is a stacked PR based on #1243.

PR Description

This PR adds a standalone CPU-side NCCL H1 probe for detecting distributed training ranks that do not enter an expected collective, or enter substantially later than their peers.

The main changes include:

• Interpose supported NCCL APIs through a Linux LD_PRELOAD library.
• Record NCCL API enter/exit boundaries without adding CUDA synchronization.
• Send probe events through a bounded CPU queue and write per-rank JSONL records from a background writer.
• Record communicator membership, communicator rank, collective sequence, API, parameters, and process identity.
• Aggregate collective records across communicator ranks in a separate node-zero analyzer.
• Detect collective_missing_enter when only part of a communicator enters an expected collective.
• Detect delayed_collective_enter when all ranks eventually enter but the arrival spread exceeds the configured threshold.
• Correlate missing ranks with the independent heartbeat component when heartbeat data is available.
• Downgrade absence-based conclusions when the probe reports dropped events.
• Integrate probe and analyzer lifecycle management into the Megatron backend, native training backend, runner, and SSH launcher.
• Write raw events and deduplicated findings as machine-readable JSONL.
• Add unit tests and real two-worker NCCL scenarios.

The H1 diagnostics distinguish between:

• A communicator rank that was not observed entering a collective.
• A rank that entered the collective substantially later than the earliest rank.
• A missing rank whose heartbeat is still alive.
• A missing rank whose heartbeat is stale and may have exited or crashed.
• A missing event whose confidence must be reduced because probe event loss is possible.

Usage

First build the Linux NCCL preload probe in a compatible build environment:

python -m flagscale.runner.tracing.build

By default, the shared library is written to:

flagscale/runner/tracing/native/libflagscale_nccl_probe.so

The library must be readable at the configured path on every training node.

Enable tracing under experiment.runner:

experiment:
  runner:
    no_shared_fs: false

    # Optional, but recommended for missing-rank liveness correlation.
    heartbeat:
      enabled: true
      publish_interval_s: 5
      process_timeout_s: 30
      progress_timeout_s: 300
      checkpoint_timeout_s: 1800

    tracing:
      enabled: true
      probe_library: /absolute/path/to/libflagscale_nccl_probe.so
      collective_timeout_s: 60
      delayed_enter_threshold_s: 30
      failure_grace_period_s: 60
      scan_interval_s: 1

After the probe has been built and tracing has been enabled, start the FlagScale training task with the existing launch command. No separate tracing command is required.

The runner automatically:

• Exports the probe through LD_PRELOAD.
• Creates a raw trace file for each training process.
• Starts one cross-rank analyzer on node 0.
• Reads heartbeat records when heartbeat monitoring is enabled.
• Writes the training exit code and preserves a bounded failure-analysis grace period.
• Stops the analyzer when the training task finishes.

Results

Tracing results are written to:

<log_dir>/tracing/<run_id>/
├── analyzer.log
├── analyzer.pid
├── findings.jsonl
├── training.exit_code
└── rank_<rank>_pid_<pid>.jsonl

The expected H1 behavior is:

• Normal collectives in which every rank enters within the threshold do not generate an H1 finding.
• If only part of the communicator enters before collective_timeout_s, a collective_missing_enter finding is recorded.
• If every rank enters but the arrival spread exceeds delayed_enter_threshold_s, a delayed_collective_enter finding is recorded.
• If the missing rank still has a fresh heartbeat, the finding reports rank_alive_but_not_entered.
• If the missing rank has a stale heartbeat, the finding reports rank_exit_or_crash_suspected.
• If the probe reports dropped events, the finding reports probe_event_loss_possible with reduced confidence.

For example:

{
  "hang_type": "collective_missing_enter",
  "run_id": "20260730_120000.000000",
  "comm_uid_hash": "0123456789abcdef",
  "comm_seq": 42,
  "detected_at_unix_ns": 1785400000000000000,
  "details": {
    "api": "ncclAllReduce",
    "expected_nranks": 4,
    "entered_comm_ranks": [0, 1, 2],
    "missing_comm_ranks": [3],
    "missing_rank_status": [
      {
        "comm_rank": 3,
        "rank": 3,
        "heartbeat": "alive",
        "probe_dropped_events": 0
      }
    ],
    "waited_s": 60.2,
    "reason": "rank_alive_but_not_entered",
    "confidence": "observed",
    "trace_event_loss_possible": false
  }
}

A missing-enter finding proves only that the analyzer did not observe the expected rank entering that collective before the timeout. It does not by itself identify whether the rank was blocked in data loading, computation, checkpointing, Python control flow, or another subsystem.

[Darryl233]

The key doesn't include "api". Is there any possibility of produce incorrect aggregation results？