Commit 465370f
authored
fix(recovery): wait after legacy container handoff (#9423)
<!-- markdownlint-disable MD041 -->
## Summary
Legacy gateway recovery now repeats the OpenShell readiness check and
managed supervisor health check after the final replacement container
restart. NemoClaw starts the primary dashboard/API host forward only
after both checks pass. Recovery stops if Docker cannot stop or start
the replacement container, or if either check fails.
### E2E claim
- Root-cause key: `gateway recovery / legacy supervisor relaunch final
handoff / post-finalize OpenShell readiness not re-established before
primary forward start`
- Source workflow: [run
`32089878527`](https://github.com/NVIDIA/NemoClaw/actions/runs/32089878527),
attempt `1`
- Failed job: `Gateway: restores the guard chain after recreation /
NVIDIA inference API key` ([job
`95570032040`](https://github.com/NVIDIA/NemoClaw/actions/runs/32089878527/job/95570032040))
- Failure signature: `Probe failed: OpenClaw gateway is running in
'e2e-2701', but the primary dashboard/API host forward could not be
re-established.`
- Claim scope: legacy managed-supervisor recovery after the final
replacement container restart.
## Related Issue
Follow-up to #9364. Issue #9364 defines the legacy compatibility
acceptance criterion. PR #9398 corrected its legacy fixture, which let
the E2E test reach this later recovery failure. This change preserves
the production supervisor allowlist.
## Changes
- Run the shared final Docker handoff function in the legacy `sleep
infinity` recovery regression test.
- Repeat the pinned managed supervisor health check after Docker
restarts the replacement container.
- Repeat the OpenShell readiness check before NemoClaw starts the
primary dashboard/API host forward.
- Stop recovery and report the failed layer when final Docker or
readiness verification fails.
- Keep prerequisite recovery failures separate from forward failures and
redact their diagnostics.
- Preserve the production supervisor allowlist.
## Type of Change
- [x] Code change (feature, bug fix, or refactor)
- [ ] Code change with doc updates
- [ ] Doc only (prose changes, no code sample modifications)
- [ ] Doc only (includes code sample changes)
## Quality Gates
- [x] Tests added or updated for changed behavior
- [ ] Existing tests cover changed behavior — justification:
- [ ] Tests not applicable — justification:
- [x] Sensitive paths changed (security, policy, credentials, preflight,
onboarding, inference, runner, sandbox, or messaging)
- [x] Sensitive-path review completed or maintainer-approved waiver
recorded — reviewer/approval link/justification: Independent security
rubric review of commit under review
`75c124bb66eadd9706da7fcef13581f93e0773e2` returned PASS with no
findings. Live E2E for this commit remains required.
- [ ] Non-success, skipped, or missing CI check accepted by maintainer —
check name, approval link, and follow-up issue:
## DGX Station Hardware Evidence
- [ ] Tested on DGX Station
- Tested commit: Not applicable
- Station profile/scenario: Not applicable
- Result: Not applicable
- Supporting evidence: Not applicable
## Verification
- [x] PR description includes a `Signed-off-by:` line and every commit
appears as `Verified` in GitHub
- [x] Normal `pre-commit`, `commit-msg`, and `pre-push` hooks passed, or
`npm run validate:pr` passed after refreshing `origin/main` when hooks
were skipped or unavailable
- [x] Targeted behavior tests pass for the current change set, or tests
are marked not applicable above — command/result or justification: `npm
exec -- vitest run --project integration
test/process-recovery-supervisor-relaunch.test.ts` (35 passed); `npm
exec -- vitest run --project cli
src/lib/actions/sandbox/connect-flow.test.ts` (40 passed); `npm exec --
vitest run --project e2e-support
test/e2e/support/launch-agent-turn.test.ts` (19 passed, 24
platform-skipped); `npm run typecheck:cli` passed; `npm run
checks:repository` passed.
- [ ] Applicable broad gate passed — `npm test` for broad
runtime/test-harness changes; `npm run check` for repo-wide
validation/coverage changes — command/result: Not applicable to this
focused recovery change; PR CI will run for latest PR commit
`75c124bb66eadd9706da7fcef13581f93e0773e2`.
- [x] Quality Gates section completed with required justifications or
waivers
- [x] No secrets, API keys, or credentials committed
- [ ] `npm run docs` builds without warnings (doc changes only)
- [ ] Doc pages follow the [style
guide](https://github.com/NVIDIA/NemoClaw/blob/main/docs/CONTRIBUTING.md)
(doc changes only)
- [ ] New doc pages include SPDX header and frontmatter (new pages only)
Documentation impact: No public documentation change. Existing recovery
documentation already describes the corrected contract.
---
Signed-off-by: Julie Yaunches <jyaunches@nvidia.com>
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Bug Fixes**
- Improved sandbox recovery failure reporting with clearer, sanitized
details.
- Prevented irrelevant manual-forward guidance when recovery fails
before forwarding.
- Preserved the most direct recovery error when multiple failure layers
are reported.
- Improved supervisor relaunch recovery with readiness checks, rollback
handling, container handoff validation, and cleanup warnings.
- Correctly reports container identity changes and managed health
failures as recovery failures.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Signed-off-by: Julie Yaunches <jyaunches@nvidia.com>1 parent 023cc46 commit 465370f
5 files changed
Lines changed: 632 additions & 116 deletions
File tree
- src/lib/actions/sandbox
- test
- support
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
907 | 907 | | |
908 | 908 | | |
909 | 909 | | |
| 910 | + | |
| 911 | + | |
| 912 | + | |
| 913 | + | |
| 914 | + | |
| 915 | + | |
| 916 | + | |
| 917 | + | |
| 918 | + | |
| 919 | + | |
| 920 | + | |
| 921 | + | |
| 922 | + | |
| 923 | + | |
| 924 | + | |
| 925 | + | |
| 926 | + | |
| 927 | + | |
| 928 | + | |
| 929 | + | |
| 930 | + | |
| 931 | + | |
| 932 | + | |
| 933 | + | |
| 934 | + | |
| 935 | + | |
| 936 | + | |
| 937 | + | |
| 938 | + | |
| 939 | + | |
| 940 | + | |
| 941 | + | |
| 942 | + | |
| 943 | + | |
| 944 | + | |
| 945 | + | |
| 946 | + | |
| 947 | + | |
| 948 | + | |
| 949 | + | |
| 950 | + | |
| 951 | + | |
| 952 | + | |
| 953 | + | |
| 954 | + | |
| 955 | + | |
| 956 | + | |
| 957 | + | |
| 958 | + | |
| 959 | + | |
| 960 | + | |
| 961 | + | |
| 962 | + | |
| 963 | + | |
| 964 | + | |
| 965 | + | |
| 966 | + | |
| 967 | + | |
| 968 | + | |
| 969 | + | |
| 970 | + | |
| 971 | + | |
| 972 | + | |
| 973 | + | |
| 974 | + | |
| 975 | + | |
| 976 | + | |
| 977 | + | |
| 978 | + | |
| 979 | + | |
| 980 | + | |
| 981 | + | |
| 982 | + | |
| 983 | + | |
| 984 | + | |
| 985 | + | |
| 986 | + | |
| 987 | + | |
| 988 | + | |
910 | 989 | | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
260 | 260 | | |
261 | 261 | | |
262 | 262 | | |
| 263 | + | |
| 264 | + | |
| 265 | + | |
| 266 | + | |
| 267 | + | |
| 268 | + | |
| 269 | + | |
| 270 | + | |
| 271 | + | |
| 272 | + | |
| 273 | + | |
| 274 | + | |
| 275 | + | |
| 276 | + | |
| 277 | + | |
263 | 278 | | |
264 | 279 | | |
265 | 280 | | |
| |||
318 | 333 | | |
319 | 334 | | |
320 | 335 | | |
| 336 | + | |
| 337 | + | |
| 338 | + | |
| 339 | + | |
| 340 | + | |
| 341 | + | |
| 342 | + | |
321 | 343 | | |
322 | 344 | | |
323 | 345 | | |
| |||
958 | 980 | | |
959 | 981 | | |
960 | 982 | | |
961 | | - | |
962 | | - | |
| 983 | + | |
| 984 | + | |
963 | 985 | | |
964 | 986 | | |
965 | 987 | | |
966 | | - | |
967 | | - | |
| 988 | + | |
| 989 | + | |
968 | 990 | | |
969 | 991 | | |
970 | | - | |
| 992 | + | |
| 993 | + | |
| 994 | + | |
| 995 | + | |
| 996 | + | |
| 997 | + | |
| 998 | + | |
971 | 999 | | |
972 | 1000 | | |
973 | 1001 | | |
| |||
0 commit comments