[DO NOT MERGE] Run-only: gb200 dsr1 measured power+temp (canonical NVIDIA)#1791
[DO NOT MERGE] Run-only: gb200 dsr1 measured power+temp (canonical NVIDIA)#1791arygupt wants to merge 19 commits into
Conversation
|
Thanks for the contribution! For vLLM & SGLang, please ensure that your recipes is similar to the official vLLM recipes and/or the SGLang cookbook If it is not, please create a PR first before we can merge your single node PR into the master branch. Let's ensure that the documentation is first class such that the entire ML community can benefit from your hard work! Thank you
PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. If re-running failed jobs is attempted, PR authors are responsible for ensuring it passes. See GitHub's docs on re-running failed jobs: https://docs.github.com/en/actions/how-tos/manage-workflow-runs/re-run-workflows-and-jobs#re-running-failed-jobs-in-a-workflow As a rule of thumb, generally, PR authors should request a review & get a PR approval from the respective companies' CODEOWNERS before requesting a review from core maintainers. If additional help is needed, PR authors can reach out to core maintainers over Slack. |
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=27578362429 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=27578777750 |
arygupt
force-pushed
the
feat/dsr1-gb200-powercheck-run
branch
from
c0b79f4 to
742665d
Compare
|
|
||
|
|
||
| # Back-compat shim — some external callers may have imported _parse_power. | ||
| _parse_power = _parse_numeric_cell |
arygupt
force-pushed
the
feat/dsr1-gb200-powercheck-run
branch
from
742665d to
06f8f37
Compare
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=27649212722 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=27650031789 |
arygupt
force-pushed
the
feat/dsr1-gb200-powercheck-run
branch
from
06f8f37 to
b357089
Compare
arygupt
force-pushed
the
feat/dsr1-gb200-powercheck-run
branch
from
b357089 to
1df33dc
Compare
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=27795211913 |
…dle days) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ronize) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…check-run # Conflicts: # .github/configs/nvidia-master.yaml # perf-changelog.yaml
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=27977727866 |
…urm@b9526e5) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=27982279578 |
…b9526e5 deployed Drop gb200+b200 from the matrix: gb200 is busy with cquil11's work (don't contend), b200 pool saturated. gb300 queue confirmed empty (squeue 36ms). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
| if ! grep -q '^health_check:' "$CONFIG_PATH"; then | ||
| printf '\nhealth_check:\n max_attempts: 540\n interval_seconds: 10\n' >> "$CONFIG_PATH" | ||
| echo "[perfmon] injected health_check (90min ceiling) into $CONFIG_PATH" | ||
| fi |
There was a problem hiding this comment.
GB300 omits health_check bump
Medium Severity
For the dsr1 perfmon path, gb200 injects a 90-minute health_check on the applied recipe before srtctl apply, but gb300 only enables monitoring on cloned recipes and never raises health_check on CONFIG_FILE. Dsr1 bring-up can exceed the default server-ready window and fail before perf samples exist.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit d6ace77. Configure here.
_0 squeue/scancel hang confirmed live: concurrent CI probe on gb300-nv_2 returned in 37ms with an empty queue while the leg on gb300-nv_0 sat 8min in pre-run Slurm cleanup. Empty queue + blocked drain = host-local slurm-client hang on _0, not a stuck job. Cancel + re-arm to land on a healthy runner. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=27987372737 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=27987975870 |
…ound cleanup timeouts Root cause of the dsr1 NVIDIA measured-power sweeps wedging ~1.5 weeks: the shared pre/post 'Slurm cleanup' anchor ran 'timeout 60 sudo rm -rf $GITHUB_WORKSPACE/benchmark_logs' on EVERY multinode runner. Only the AMD path (launch_mi355x-amds.sh) creates benchmark_logs; NVIDIA launchers never do. On the GB300 login host sudo hangs resolving SSSD policy (nsswitch 'sudoers: files sss') BEFORE exec'ing rm, and 'timeout 60' only sends SIGTERM — which the stuck root sudo ignores — so timeout waits in sigsuspend forever and the step never returns. Live process table (probe 27988714413) showed timeout/sudo pairs for gharunner0/1/2 stuck 4+ DAYS. The command exists only on feature branches (added 475ce8a, PR #1574); not on main — which is why only this campaign's branch wedged. Fix: (1) run the privileged benchmark_logs cleanup only on AMD (case mi*) runners, with 'timeout --kill-after=5s 60s sudo -n' so a hung sudo is force-KILLed and never prompts; (2) harden every Slurm call with 'timeout --kill-after=5s 30s' (TERM then KILL) and break the drain loop explicitly on squeue timeout/failure, so no cleanup command can block indefinitely. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…d to AMD) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…bound Slurm calls Prevents the NVIDIA wedge from re-entering main when this PR lands. The shared cleanup ran 'sudo rm -rf benchmark_logs' on every multinode runner, but only the AMD path (launch_mi355x-amds.sh) creates that dir. On GB300 login hosts sudo hangs resolving SSSD policy before exec'ing rm, and the bare 'sudo rm' (no timeout) hangs forever. Scope it to AMD (case mi*) with 'timeout --kill-after=5s 60s sudo -n', and hard-bound every Slurm call (TERM-then-KILL) with an explicit loop break on squeue timeout/failure. Mirrors the fix validated live on PR #1791 (run 27989491427 cleared cleanup on gb300-nv_1; probe 27988714413 showed 4-day-stuck timeout/sudo pairs). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=27989491427 |
…dated) gb300 powercheck passed (run 27989491427) + graphed into the canvas. Arm the remaining canonical NVIDIA multinode platforms: gb200 (1P TP4 + 2D TP4) and b200 (1P TP4 + 5D TP8 = 44 GPU). Same perfmon wiring; unblocked by the sudo/SSSD cleanup fix (94c2add, mi*-scoped). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
| _csv_arg = next( | ||
| (Path(p) for p in _csv_candidates if p and Path(p).is_file()), | ||
| None, | ||
| ) |
There was a problem hiding this comment.
Multinode misses stale CSV guard
Medium Severity
When perfmon staging finds no CSVs, launchers omit GPU_METRICS_CSV_GLOB, so process_result.py still uses the single-node gpu_metrics.csv fallback. That contradicts the multinode branch that refuses stale single-CSV data when the glob env var is set but matches nothing.
Reviewed by Cursor Bugbot for commit c8d2e5d. Configure here.
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
There are 3 total unresolved issues (including 2 from previous reviews).
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 35e02d1. Configure here.
| awk '/^base:/{print; print " monitoring:"; print " enabled: true"; print " sample_interval: 1.0"; next} {print}' "$CFG" > "$CFG.perfmon.tmp" && mv "$CFG.perfmon.tmp" "$CFG" | ||
| echo "[perfmon] injected monitoring: under base: in $CFG" | ||
| fi | ||
| sed -i 's/^ max_attempts: [0-9]*/ max_attempts: 540/' "$CFG" |
There was a problem hiding this comment.
b200 sed overwrites all retries
Low Severity
Under PERFMON_ENABLED, sed replaces every line matching ^ max_attempts: in the recipe file with 540, not only base.health_check.max_attempts. Any other four-space max_attempts field in the same YAML is forced to the same value.
Reviewed by Cursor Bugbot for commit 35e02d1. Configure here.
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=27991175106 |
…eaped on watchtower) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=27993697873 |
1 participant
Purpose: get canonical NVIDIA measured dsr1 power+temp on the healthy gb200 runner, sidestepping the wedged gb300-nv fleet.
Carries #1574's self-contained consumer code (
aggregate_power.py/process_result.py+ tests) on top ofmain, plus the gb200 launcher perfmon wiring and a single 1-job changelog entry (dsr1-fp4-gb200-dynamo-sglang-powercheck). Matrix = exactly one gb200 job (1k/1k, conc 8, 1×prefill TP4 + 2×decode TP4) — verified withgenerate_sweep_configs.Same proven pipeline as the gb300 CoreWeave validation (run
27170607165):launch_gb200-nv.shclones the perfmon fork (gb200 dsr1 recipes + perfmon machinery), injectsmonitoring:+ a 90-minhealth_check, and stagesperf_samples_*.csv+GPU_METRICS_CSV_GLOB→ Process result →aggregate_power.pypatches the agg JSON withavg_power_w+ prefill/decode + per-worker temp/util/mem.Why not merge: duplicates #1574's
aggregate_power.py. Closed once the data lands.Known risk to watch on first run: the Oracle/watchtower gb200 cluster may not cross-mount compute→head, so per-node
perf_samples(written on compute) may need the shared-FS treatment the minimax path already uses (flagged inline in the launcher).🤖 Generated with Claude Code
Note
Medium Risk
Large changes to published agg JSON fields and multinode CI pre-run behavior; run-only configs duplicate #1574 consumer logic but still affect benchmark ETL inputs when executed.
Overview
Adds run-only
*-powerchecksweep entries (gb200 / b200 / gb300) so a single dsr1 disagg job can validate measured board power and GPU telemetry before a full NVIDIA campaign.CI / runners: Multinode pre-run Slurm cleanup now uses bounded
timeout/scanceland skips thesudo rm benchmark_logsstep on NVIDIA hosts (AMDmi*only), avoiding hangs on SSSD-backed sudo. For dsr1 on gb200/gb300/b200 launchers, the flow switches toSemiAnalysisAI/srt-slurm@feat/inferencex-perfmon, injects recipemonitoring:(and longer health checks where needed), copiesperf_samples_*.csvinto the workspace, and setsGPU_METRICS_CSV_GLOBbeforeoutputscleanup.Result pipeline:
process_result.pyprefers the glob over single-node CSVs (no stale fallback when glob is set) and passesdisagginto aggregation.aggregate_power.pygrows to merge many per-node CSVs (GPU index namespacing), parse perfmon filenames intoworkers[], add temp/util/mem fields, support multinode bench windows viadate+duration, and apply per-stage joules when--disaggand both prefill/decode workers exist.Reviewed by Cursor Bugbot for commit 76bbfcb. Bugbot is set up for automated code reviews on this repo. Configure here.