| title | Recover and Rebuild Sandboxes | ||||
|---|---|---|---|---|---|
| sidebar-title | Recover and Rebuild Sandboxes | ||||
| description | Recover a stopped agent runtime or rebuild a sandbox while preserving supported state. | ||||
| description-agent | Explains the supported recovery and rebuild paths for OpenClaw, Hermes, and Deep Agents sandboxes. Use when a gateway or terminal runtime is degraded or when a sandbox image must be recreated. | ||||
| keywords |
|
||||
| content |
|
||||
| skill |
|
Use the lightest recovery operation that repairs the sandbox while preserving its supported state.
If NemoClaw reports that a Docker-driver sandbox is stopped, restart the existing container:
$$nemoclaw <sandbox-name> startThis path preserves the sandbox workspace and repairs the agent runtime and host-side forwards after the container starts.
If the container is paused, follow the printed docker unpause guidance instead.
If Docker no longer has the container, follow the printed rebuild --yes guidance so NemoClaw can recreate the sandbox from its recorded metadata.
If the sandbox has shields up and the OpenClaw gateway does not start after the container restarts, lower shields before you rebuild:
$$nemoclaw <sandbox-name> shields downWhile holding the config mutation lock, NemoClaw confirms that no startup process runs and no readiness lease exists.
Only then does it accept shields down.
Other locked-config operations still require the lease.
This failed-startup recovery path requires a sandbox image that includes the in-container OpenClaw config and state guards.
If NemoClaw reports that the config guard is absent, upgrade the CLI.
Then rebuild the sandbox before you retry recovery.
After shields are down, start the sandbox again.
When the OpenClaw gateway is healthy, rerun shields up.
$$nemoclaw <sandbox-name> recoverThe command repairs a stopped in-sandbox gateway and re-establishes the dashboard port-forward in one step.
It is idempotent and safe to script.
If the gateway is already healthy, recover exits after the probe and does not restart it.
If the host forward is already active, recovery accepts it only after OpenShell ownership is reconciled and the local endpoint is reachable.
Use gateway restart when you intentionally need a supported OpenClaw gateway to reload runtime configuration or plugins.
$$nemoclaw <sandbox-name> gateway restartThe restart command asks the topology-specific controller to stop the tracked gateway child, wait for the entrypoint to launch a replacement, and prove listener and HTTP health.
The host then checks or recovers host-side dashboard, messaging, and agent forwards.
Refer to $$nemoclaw <name> recover and $$nemoclaw <name> gateway restart for details.
$$nemoclaw <sandbox-name> recoverThe command repairs a stopped in-sandbox gateway and re-establishes the dashboard port-forward in one step.
It is idempotent and safe to script.
If the gateway is already healthy, recover does not restart it.
If the host forward is already active, recovery accepts it only after OpenShell ownership is reconciled and the local endpoint is reachable.
Before it repairs the gateway, recover checks for a NemoClaw cron restore gate or release recovery record left by an interrupted rebuild.
The gate continues to block new Hermes turns and cron dispatch across gateway and container restarts in the same sandbox.
If release rollback could not restore the gate, recover uses the root-owned recovery record to reacquire it before gateway repair can start dispatch.
After gateway repair, recover waits for active agent work to finish and validates the restored cron jobs and scripts.
It clears NemoClaw-owned gate and release recovery state only after validation succeeds.
If no independent operator drain exists, successful recovery prints Hermes cron dispatch resumed after restored jobs and scripts were validated.
If an operator drain exists, recovery prints Hermes cron restore gate cleared; the independent operator drain remains active.
The command does not own or clear the Hermes operator drain, so new Hermes turns and cron dispatch remain blocked while that drain is active.
If gate reacquisition or cron validation fails, recover exits nonzero and retains the recovery state for another attempt.
Use gateway restart when you intentionally need a supported Hermes gateway to reload runtime configuration or plugins.
$$nemoclaw <sandbox-name> gateway restartThe restart command asks the topology-specific controller to stop the tracked gateway child, wait for the entrypoint to launch a replacement, and prove listener and HTTP health. The host then checks or recovers host-side dashboard, messaging, and agent forwards.
For Hermes, the entrypoint supervisor owns the gateway, dashboard process, internal API relay, dashboard relay, and gateway log stream. The nonroot managed supervisor repairs those processes continuously, stops an alive but deaf gateway after four consecutive failed health checks, and quarantines relaunch after five exits within 60 seconds until sandbox recreation.
The host does not start the in-sandbox processes independently.
Refer to $$nemoclaw <name> recover and $$nemoclaw <name> gateway restart for details.
NemoClaw keeps the previous container available throughout the replacement health, OpenShell readiness, state restoration, gateway restart, and settle checks.
Before recreation, NemoClaw backs up the state directories and files declared by the agent manifest.
NemoClaw waits for OpenShell to re-register the exact replacement before it restores state.
After state restoration, it restarts the gateway in that replacement and reruns the managed health and settle checks.
It commits only after the replacement identity, state restoration, gateway restart, and settle check pass.
NemoClaw removes the temporary state backup after a successful restore or rollback.
If state restoration and rollback both fail, it retains the backup and prints host recovery guidance.
Mounted state remains available, but a committed swap does not retain other writable-layer changes.
After a transactional recreation, NemoClaw waits 120 seconds for OpenShell to re-register the sandbox before state restoration and replacement commit.
Set NEMOCLAW_GATEWAY_RECOVERY_WAIT_SECONDS before the recovery command to change this budget.
A definitive managed-health failure still stops immediately.
If re-registration, state restoration, or a later gateway check fails, NemoClaw attempts to roll back the replacement and leaves the primary dashboard or API host forward stopped.
For the controller topology, trust boundary, and fail-closed conditions, refer to Understand Gateway Lifecycle Control.
If recovery cannot repair a sandbox that needs credentials or a current controller contract, rebuild it.
Deep Agents sandboxes are terminal runtimes and do not expose an OpenClaw or Hermes in-sandbox gateway.
Use nemo-deepagents <sandbox-name> status, logs, connect, and rebuild for recovery.
If the terminal runtime reports degraded health, rebuild the sandbox instead of using recover or gateway restart.
A successful complete preflight for $$nemoclaw launch <sandbox-name> can publish a credential-free launch-readiness lease on Linux.
Linux infrastructure can publish the same evidence with $$nemoclaw <sandbox-name> connect --probe-only.
The lease has a fixed 24-hour lifetime that repeated launches do not extend.
Leaving the agent with /exit does not revoke it, and users do not refresh it manually.
During the lease, launch still verifies the owning OpenShell gateway, exact live sandbox identity, registry and agent configuration, effective policy, inference route, required forwards, and semantic runtime health.
Configured inference must return HTTP 2xx from the semantic inference.local probe, which is stricter than the HTTP 200–499 reachability diagnostic used by ordinary connect.
For OpenClaw, connect --probe-only settles the existing allowlisted pairing flow before it publishes a credential-free pairing qualification with the lease.
The readiness evidence binds the OpenClaw version and trusted registry and agent manifest configuration.
Its pairing qualification binds the canonical CLI client, exact paired device identity, required operator role and scopes, owning OpenShell gateway, sandbox lifecycle identity, and fixed lease epoch.
Before lease acceptance, launch makes a bounded, read-only observation of the current OpenClaw-owned pairing state through the owning OpenShell gateway.
It skips the complete pairing approval pass only when the qualification still matches exactly and no relevant allowlisted request is pending.
Any missing, unreadable, malformed, ambiguous, or changed observation runs the complete pairing approval pass.
A relevant allowlisted pending request also runs that complete path.
For missing, unsafe, malformed, expired, mismatched, changed, or unhealthy evidence, NemoClaw fences any prior acceptable evidence before it runs the complete preflight.
Ordinary launch continues only when NemoClaw proves that no old authority or evidence can exist, or durably rotates the runtime epoch.
If an old epoch might exist and cannot be durably rotated, launch and connect --probe-only stop before complete preflight or recovery.
Their redacted guidance asks you to repair the current user's secure OS runtime authority and NemoClaw state permissions, then retry.
If NemoClaw securely proves that both the authority and receipt are absent but cannot create new authority, ordinary launch can run the complete preflight without optimization; connect --probe-only exits nonzero because it could not publish evidence.
If that preflight succeeds before the lease expires, replacement evidence keeps the original start and expiry time.
After expiry, a successful complete preflight starts a new 24-hour lease only when publication succeeds.
Before the first mutation in the complete preflight, the producer revalidates its sandbox-global runtime epoch under the sandbox lifecycle lock followed by the owning gateway lock. It holds both locks through all mutations in the complete preflight, final state capture, and publication. A stale producer makes no changes and re-inspects the newer lease.
If unsafe or malformed authority history makes the prior lease timeline untrustworthy, NemoClaw durably invalidates the old epoch and starts one conservative 24-hour quarantine. Publication remains disabled until both wall time and monotonic uptime span the full quarantine. Repeated attempts do not extend it. After it elapses, the next successful complete preflight can publish a new fixed 24-hour lease.
Lease acceptance and publication are currently Linux-only and require a secure, independently writable OS per-user runtime authority under /run/user/<numeric-uid>.
It never uses caller-provided environment variables to select this authority.
On macOS, launch runs the complete preflight every time and does not publish a launch-readiness lease.
connect --probe-only also runs the complete preflight, including recovery and probes, but exits nonzero because it cannot publish authoritative launch-readiness evidence.
The publication-failure diagnostic is redacted and does not print filesystem paths or environment values.
Infrastructure must run connect --probe-only as the same final numeric user that later runs launch.
Run it after the final durable home and state volume is mounted and after policy and network provisioning is complete.
NemoClaw rejects evidence after a bound sandbox, configuration, policy, or network identity changes.
Deployment ordering remains responsible for external changes that OpenShell and NemoClaw cannot observe.
recover and start repair the in-sandbox agent gateway and the host-side port forwards for one named sandbox.
connect --probe-only waits for that sandbox to become ready, rechecks it on its recorded gateway, and then verifies or repairs the same sandbox-scoped processes and forwards.
These commands do not restart, replace, or reap the shared host OpenShell gateway process.
If the host gateway RPC returns an error while these sandbox-scoped commands run, they surface the error with explicit next-step guidance and exit rather than swapping the shared gateway out from under other sandboxes.
To repair the host gateway itself, use openshell status, openshell gateway start --name <gateway>, or run $$nemoclaw onboard again.
Other workflows, including onboarding, rebuild, and doctor --fix, can explicitly recover the named host gateway when their operation requires it.
$$nemoclaw <name> gateway restart instead restarts only the supported agent gateway inside the named sandbox.
For a portable experimental-profile sandbox with the recorded ollama-local provider, connect --probe-only and recover also verify the host-side inference chain.
Before it decides whether to start Ollama, the command probes http://127.0.0.1:11434/api/tags and leaves a healthy daemon unchanged.
When that API is unhealthy, it starts the fixed user-local executable only if NemoClaw has a valid ownership receipt.
NemoClaw releases that predate this receipt do not claim an existing executable after an upgrade.
To authorize recovery for a previous NemoClaw user-local install, first verify that ${HOME}/.local/bin/ollama is the executable you want NemoClaw to manage.
Then run NEMOCLAW_PORTABLE_OLLAMA_REENROLL=1 nemoclaw <name> recover once.
The command rejects a symbolic link or non-executable file before it records ownership.
It refuses to launch a duplicate when another ollama process exists but the API remains unhealthy.
After a launch, recovery waits up to 30 seconds for /api/tags to return valid JSON with a models array.
It does not take over a system service or an unrelated user-managed daemon.
It refuses a symbolic link, non-regular file, or non-executable file at the receipt-bound executable path.
On every ollama-local completion path, the command verifies the authenticated proxy on port 11435.
It also requires HTTP 2xx from https://inference.local/v1/models before it reports success.
If Ollama does not become healthy within 30 seconds, the command identifies the exact receipt-bound executable and its serve argument, then tells you to retry recovery.
An Ollama startup or route failure exits non-zero and prints the available recovery guidance.
$$nemoclaw <sandbox-name> rebuildBefore it backs up or deletes the existing sandbox, rebuild validates the recorded sandbox, gateway, policy, MCP, agent, and operation-lock state.
When one of these checks fails, NemoClaw prints Rebuild preflight failed, explains how to recover, and ends with Aborting rebuild.
At this boundary, the existing sandbox is unchanged and no sandbox data has been removed.
Use the recovery guidance that matches the reported check:
- Verify the sandbox name when its registry entry is missing.
- Follow the printed OpenShell gateway recovery steps when the gateway schema is incompatible.
- Repair the named pending baseline policy transition, then rerun
rebuild. - Resolve an incomplete MCP destroy transaction before retrying.
- Back up the sandbox state and recreate it with
$$nemoclaw onboardwhen the record contains multiple agents. Transactional multi-agent rebuild is not supported. - Wait for another onboarding or rebuild operation to finish before retrying. If verified stale-lock cleanup is still in progress, wait briefly and rerun the command. Do not delete the lock manually.
$$nemoclaw <sandbox-name> gateway-token --quietBefore post-restore repairs, NemoClaw verifies that the recreated sandbox still identifies as Hermes and exits nonzero if its identity does not match the rebuild target.
After state restore, NemoClaw restarts the Hermes gateway so it reads the restored durable state, then restores managed MCP configuration through the normal lifecycle.
MCP restoration performs an acknowledged gateway reload, so NemoClaw finishes by verifying the final running gateway and its managed MCP state without replacing that verified process again.
The gateway starts during recreation and reads its durable state before the restore replaces it, which is why the first post-restore restart must happen before managed MCP restoration.
rebuild exits nonzero instead of reporting success when it cannot verify final gateway health or managed MCP state.
Follow the printed recovery guidance, using $$nemoclaw <sandbox-name> gateway restart first for gateway health, $$nemoclaw <sandbox-name> recover when the restart does not restore verified health, and $$nemoclaw <sandbox-name> mcp restart for incomplete managed MCP restoration.
When the rebuild backup contains active Hermes cron jobs that reference scripts, NemoClaw validates those script references before it deletes the existing sandbox.
The check covers the default profile and named profiles.
Each referenced script must exist, be readable, be a regular file, and remain inside its profile's scripts directory.
Disabled and paused jobs do not require their referenced scripts.
If this validation fails, the rebuild keeps the existing sandbox and reports the preserved backup path.
After NemoClaw creates the replacement, it acquires an independent root-owned gate that blocks new Hermes turns and cron dispatch.
The gate remains active across gateway and container restarts in the replacement sandbox.
NemoClaw waits for active agent work to finish before restoring state.
It validates the restored jobs and scripts before the gateway replacement, then keeps dispatch blocked while it restarts and verifies that replacement.
It records the replacement process identity around managed health verification and clears the gate only if that same live process completes the final cron validation.
If an operator already drained the gateway, NemoClaw clears its gate and release recovery record while leaving the operator drain active.
If state restore, managed MCP restoration, gateway replacement, or cron validation fails after gate acquisition, the command exits nonzero and preserves the backup.
Those failures retain the NemoClaw gate unless the output explicitly reports that release rollback could not restore its marker.
In that exceptional case, NemoClaw preserves a root-owned release recovery record, but you must not assume dispatch is blocked.
Run $$nemoclaw <sandbox-name> recover immediately so it can reacquire the gate before validating the restored cron state.
If gate reacquisition fails, recovery exits nonzero and leaves the recovery record in place for another attempt.
Failures before gate acquisition do not create a new gate.
Do not manually remove the root-owned cron restore marker or release recovery record because removal bypasses restored cron validation.
If managed MCP restoration failed, correct the reported cause and run $$nemoclaw <sandbox-name> mcp restart first.
Then run $$nemoclaw <sandbox-name> recover to repair and probe the gateway, validate the restored cron tree, and clear NemoClaw-owned cron restore recovery state.
Before rebuild deletes the existing sandbox, NemoClaw records a replacement journal in the onboarding session.
The journal binds the operation to the sandbox name, recorded OpenShell gateway, source identity, and replacement settings.
It stores fingerprints instead of credential values or raw OpenShell sandbox IDs.
If rebuild stops after recording the journal, rerun the command with the same replacement settings.
The rerun takes one of these actions:
- It continues deletion when the live sandbox still has the journaled source identity.
- It continues creation when the recorded OpenShell gateway explicitly reports the source sandbox as absent.
- It accepts an existing replacement only when its live identity and sandbox registry generation match the journal.
An accepted replacement is not deleted again.
A rerun that accepts a journaled Hermes replacement checks for any retained NemoClaw gate before it retires the replacement journal.
When the gate exists, the rerun validates the restored cron tree and releases the gate first.
If validation or gate release fails, NemoClaw preserves the accepted replacement, state backup, and replacement journal.
New Hermes turns and cron dispatch remain blocked.
Correct the reported restore problem, then run $$nemoclaw <name> recover.
After recovery succeeds, rerun rebuild with the same replacement settings so NemoClaw can retire the journal.
The command reports Sandbox '<name>' already holds the replacement from the interrupted rebuild. and preserves the state backup path when one exists.
Pass --verbose to include the replacement identifier, OpenShell gateway, and journal phase in rebuild diagnostics.
After the sandbox registry proves the journaled replacement identity and generation, NemoClaw removes an obsolete source image that it owns.
It retains the image when the source is shared or the registered replacement reuses it.
If image removal fails, NemoClaw keeps the accepted replacement and tells you to run $$nemoclaw gc for cleanup.
NemoClaw fails closed when the selected gateway, replacement settings, durable source registry fields, or live source or target identity no longer matches the journal. The error names the sandbox and the mismatch that stopped recovery. Do not delete a same-name sandbox to bypass this check. Inspect the named OpenShell gateway and sandbox, correct the reported drift, and rerun the original command.
A same-name recreation started by $$nemoclaw onboard uses the same replacement journal.
If that recreation is interrupted after the Journaled replacement message, rerun the original onboarding command with the same target settings.
The active replacement can continue without adding --resume.
Use --resume for interrupted onboarding steps that occur before a replacement journal exists.
If an archive command preserves at least one state directory, NemoClaw keeps the usable entries and reports the manifest-defined paths that could not be archived.
If a manifest-declared state file fails, NemoClaw stops before deleting the original sandbox even when it preserved state directories, unless you explicitly pass --force.
If every state directory fails, NemoClaw stops before deleting the original sandbox even when it captured loose files, unless you explicitly pass --force.
When rebuild starts with shields up, NemoClaw opens a 30-minute shields-down window for backup and recreation. A detached auto-lock timer remains the recovery authority until NemoClaw commits a successful shields-up state, including when the host rebuild process exits unexpectedly.
If a failed shields transition on a sandbox from an older NemoClaw release quarantined the OpenClaw config, the bytes are preserved as `/sandbox/.openclaw/.nemoclaw-rejected-openclaw.json-` rather than deleted. Upgrade the NemoClaw CLI before rebuilding because an older CLI restages the older in-container guard. To preserve settings, copy the quarantine file out of the container before `rebuild --yes`. After the rebuild, inspect that copy and reapply required settings with the host-side `config set` command. To discard the quarantined settings, upgrade the CLI and run `rebuild --yes` to create a known-good baseline. Sandboxes with the updated guard report quarantine filenames and synthesize a missing `.config-hash` only during `shields up` from the default mutable posture. For an older Hermes image that predates sealed shields transitions, only the rebuild workflow may use the descriptor-safe compatibility transition needed to archive and replace the sandbox. That transition verifies the strict and compatibility hashes and publishes fresh config inodes before changing their lock posture, while ordinary `shields up` and `shields down` commands continue to refuse the older protocol.Refer to $$nemoclaw <name> rebuild for flag details.
- Use Switch Inference Providers to change a model or provider.
- Create and Restore Snapshots for the state-preservation contract.
- Troubleshooting for
privileged control unavailable, stopped sandboxes, and failed rebuilds.