[No Commit] Investigating B200 CI: expand NVIDIA state dump and add CUDA-check retry

[AmesingFlank]

Summary

• All 18 B200 jobs on the 2026-07-08 nightly (run 28930002457) failed at infra steps: 11 at CUDA Compute Check with CUDA unknown error, 7 at Initialize containers with all predefined address pools have been fully subnetted. Docker-network failures are localized to a single host (dgxb200-03); the CUDA failures span dgxb200-04 and dgxb200-05.
• The current CUDA-check log tells us only that torch.cuda.is_available() returned False. We cannot see whether UVM device nodes are missing, fabric-manager socket is down, CUDA_VISIBLE_DEVICES got clobbered, or something else. This PR expands the pre-check state dump and wraps the check with a retry + failure forensics so the next occurrence self-documents.
• Diagnostic-only — no behavior change on healthy runners. The [No Commit] title is intentional; this is meant to keep on diag-b200-cuda until we have signal from a nightly run.

What changes

• Run NVIDIA command: dumps /dev/nvidia*, nvidia kernel modules, NVIDIA_/CUDA_ env vars, and the fabric-manager socket next to nvidia-smi. Runs before the CUDA check so we get the state even when the check fails.
• CUDA Compute Check: calls torch.cuda.init() explicitly and prints the traceback (currently silenced as a UserWarning), retries once after 15s, and dumps /dev/nvidia* + nvidia-smi between/after attempts. The retry also covers the observed pattern where the same runner host succeeded on 2026-07-07 and failed on 2026-07-08 — a transient host-side race is a plausible cause.

Test plan

• YAML parses (checked locally with yaml.safe_load)
• Bash -n syntax check on both step scripts passes (heredoc terminator lands at column 0 after YAML normalization)
• Wait for a nightly run to land; confirm we see the extra state dumped on any failing B200 job
• If runner infra is healed by then, kick a manual benchmark_dispatch on B200 to prove the diagnostic wiring executes cleanly on a healthy runner

🤖 Generated with Claude Code

[AmesingFlank]

Extended this PR to mirror both diagnostic edits into .github/workflows/test.yml. That's the only PR-triggered workflow that lands on linux.dgx.b200, so the new state dump + CUDA-check retry now execute automatically on every push to this branch — no manual dispatch needed.

Also closed #3019: the host-probe workflow was workflow_dispatch-only and I don't have dispatch permission on this repo, so it would never have produced any signal. When we can see actual failing output from test.yml's Run NVIDIA command + CUDA Compute Check here, that will tell us most of what a host probe would have told us anyway (missing UVM node, fabric-manager socket, env clobber).

Skipped the second Run NVIDIA command in test.yml (line ~425, test-notebooks job) because that job runs on linux.g5.4xlarge.nvidia.gpu (A10G), which is not part of the affected B200 pool.