troubleshooting.mdx

title	Troubleshooting
sidebar-title	Troubleshooting
description	Diagnose and resolve common NemoClaw installation, onboarding, and runtime issues.
description-agent	Lists fixes for common installation, onboarding, and runtime issues. Use when diagnosing a reported NemoClaw error, a failed onboard, or unexpected sandbox behavior.
keywords	nemoclaw troubleshooting nemoclaw debug sandbox issues openclaw tool calling raw tool call json
content	type reference

{/* markdownlint-disable MD014 */}

This page covers common installation, onboarding, and runtime issues, along with resolution steps.

The diagnostic commands on this page assume `$$nemoclaw` is on your `PATH` (re-source your shell profile after an nvm- or fnm-managed install) and that your user can reach the Docker socket — either as a member of the `docker` group or by running the Docker commands with `sudo`.

If your issue is not listed here, join the NemoClaw Discord channel to ask questions and get help from the community. You can also file an issue on GitHub.

Installation

$$nemoclaw not found after install

If you use nvm or fnm to manage Node.js, the installer may not update your current shell's PATH. The $$nemoclaw binary is installed but the shell session does not know where to find it.

Run source ~/.bashrc (or source ~/.zshrc for zsh), or open a new terminal window.

When installing from a source checkout with npm install, NemoClaw first tries npm link. If the global npm prefix is not writable, it writes a managed shim to ~/.local/bin/nemoclaw instead. Add ~/.local/bin to your PATH if the command is still not found. Source-checkout installs also bootstrap OpenShell when it is missing before running preflight. If a source install still reports that openshell is not available, re-run the installer from the repository root and check that ~/.local/bin is on your PATH.

Installer fails on unsupported platform

The installer checks for a supported OS and architecture before proceeding. If you see an unsupported platform error, verify that you are running on a tested platform listed in the Container Runtimes table in the quickstart guide.

Node.js version is too old

NemoClaw requires Node.js 22.19 or later. If the installer exits with a Node.js version error, check your current version:

node --version

If the version is below 22.19, install a supported release. If you use nvm, run:

nvm install 22
nvm use 22

Then re-run the installer.

Contributor Setup Fails with a JavaScript Heap Out-of-Memory Error

This applies to a source checkout, not to an installed release.

Node.js derives its default old-space limit from host memory. On a host with 8 GB of RAM, that limit is about 2.2 GB. The CLI type check needs more heap than that limit, so ./scripts/dev-setup.sh stops at the type-check step and Node.js reports JavaScript heap out of memory.

Raise the limit, then run setup again:

export NODE_OPTIONS=--max-old-space-size=5120
./scripts/dev-setup.sh

Keep that variable set for later type-check, build, and test commands.

Image push fails with out-of-memory errors

The sandbox image is approximately 2.4 GB compressed. During image push, the Docker daemon, k3s, and the OpenShell gateway run alongside the export pipeline, which buffers decompressed layers in memory. On machines with less than 8 GB of RAM, this combined usage can trigger the OOM killer.

If you cannot add memory, configure at least 8 GB of swap to work around the issue at the cost of slower performance.

Docker is not running

Check the host before onboarding:

$$nemoclaw host probe

The command does not start Docker or apply a repair. A host.docker.daemon_unreachable finding means Docker is installed but NemoClaw cannot reach the daemon. For JSON output and exit-code details, refer to System Readiness.

The installer and onboard wizard require Docker to be running. If you see a Docker connection error, start the Docker daemon:

sudo systemctl start docker

On macOS with Docker Desktop, open the Docker Desktop application and wait for it to finish starting before retrying.

Docker permission denied on Linux

On Linux, if the Docker daemon is running but you see "permission denied" errors, your user may not be in the docker group. The installer can add your user to the group, but Linux does not activate that membership in the current shell automatically. Add your user and activate the group in the current shell:

NemoClaw needs Docker access. On personal Linux development machines, adding your user to the `docker` group is the standard way to run Docker without sudo. Members of the `docker` group can control the daemon with root-level impact, so grant this access only to trusted local accounts; on shared or managed systems, use your organization's approved Docker access path. For background, review Docker's [daemon attack surface guidance](https://docs.docker.com/engine/security/#docker-daemon-attack-surface).

sudo usermod -aG docker $USER
newgrp docker

Then retry $$nemoclaw onboard. If the installer stopped after printing newgrp docker, run that command and then re-run the installer:

newgrp docker
curl -fsSL https://www.nvidia.com/nemoclaw.sh | bash

Installer reports Docker access outside the docker group

On Linux, the installer may report that Docker is reachable even though your user is not in the docker group. This means the host grants Docker daemon access through another path, such as a custom DOCKER_HOST, socket ACL, or managed runtime policy. NemoClaw can continue when docker info works, but the diagnostic explains why a negative Docker-permission test will not reproduce on that host.

Check the Docker access path before relying on the host as a clean permission baseline:

id -nG
echo "${DOCKER_HOST:-}"
docker info

The managed default gateway service accepts DOCKER_HOST only as an absolute local unix:// socket path. It rejects remote endpoints and relative socket paths before service startup.

Onboarding Reports an Invalid Docker Host

The invalid_docker_host advisory means that DOCKER_HOST is not an absolute local unix:// socket path that NemoClaw can write to the managed OpenShell gateway service environment. NemoClaw does not use the standalone gateway fallback when this validation fails. Onboarding prints the advisory identifier in parentheses after each action title in the Suggested fix list. The terminal output names invalid_docker_host when this validation fails, so you can match the message to this section. Remove the override to use Docker's default local socket:

unset DOCKER_HOST
$$nemoclaw onboard

If your Docker daemon uses another local socket, set an absolute unix:// path before you retry:

export DOCKER_HOST=unix:///var/run/docker.sock
$$nemoclaw onboard

NemoClaw rejects TCP and SSH endpoints, relative socket paths, values that contain single quotes, and values that contain line breaks. Do not wrap the socket path in single quotes inside the variable value.

macOS first-run failures

The two most common first-run failures on macOS are missing developer tools and Docker connection errors.

To avoid these issues, install the prerequisites in the following order before running the NemoClaw installer:

1. Install Xcode Command Line Tools (xcode-select --install). These are needed by the installer and Node.js toolchain.
2. Install and start a supported container runtime (Docker Desktop or Colima). Without a running runtime, the installer cannot connect to Docker.

docker is missing after installing Colima

Homebrew Colima does not install the Docker CLI binary. If you install only Colima, colima start can succeed while later docker commands fail with command not found.

Install both packages, start Colima with enough resources for the sandbox image build, and verify Docker before onboarding:

brew install colima docker
colima start --cpu 4 --memory 8
docker info

Permission errors during installation

The NemoClaw installer does not require sudo or root. It installs Node.js via nvm and NemoClaw via npm, both into user-local directories. The installer also handles OpenShell installation automatically using a pinned release.

If you see permission errors during installation, they typically come from Docker, not the NemoClaw installer itself. Docker must be installed and running before you run the installer, and installing Docker may require elevated privileges on Linux.

npm install fails with permission errors

If npm install fails with an EACCES permission error, do not run npm with sudo. Instead, configure npm to use a directory you own:

mkdir -p ~/.npm-global
npm config set prefix ~/.npm-global
export PATH=~/.npm-global/bin:$PATH

Add the export line to your ~/.bashrc or ~/.zshrc to make it permanent, then re-run the installer.

Installer fails on NVIDIA Jetson

The installer auto-detects NVIDIA Jetson devices (Orin and Thor) and applies required host configuration before the normal install flow. If the Jetson setup step fails, verify that you have sudo access and that Docker is installed and running.

For JetPack 6 (L4T 36.x), the setup switches iptables to legacy mode and adjusts the Docker daemon configuration. For JetPack 7 (L4T 38.x / Thor), only bridge netfilter and sysctl settings are applied. For JetPack 7 (L4T 39.x), bridge netfilter is loaded only when the host is missing it. Some R39 images already ship with br_netfilter configured and are left untouched. On affected R39 hosts, the installer prints loading br_netfilter (required by k3s inside the OpenShell gateway). Without this fix, sandbox pods fail DNS resolution against the in-cluster service and the onboard Setting up OpenClaw inside sandbox step times out.

If the L4T version is not recognized, the setup step is skipped and the installer continues normally.

DNS resolution from inside docker fails (corporate firewall)

Some corporate networks block outbound UDP port 53 to public DNS servers and force all host name resolution through DNS over TLS on TCP port 853. Containers do not inherit the host's DNS-over-TLS configuration, so the sandbox build's npm ci step times out trying to resolve registry.npmjs.org against 1.1.1.1 or 8.8.8.8.

NemoClaw's preflight runs a short docker run --rm busybox nslookup nemoclaw-dns-probe-<random>.invalid probe before starting the sandbox build. The fresh .invalid name should return NXDOMAIN through a working resolver, so cached answers cannot hide blocked DNS egress. When the probe confirms a DNS failure, onboarding stops with platform-specific remediation instead of hanging for ~15 minutes and printing a cryptic Exit handler never called.

Use the preflight headline to choose the recovery path:

• If no DNS servers could be reached, Docker could not reach its configured resolver. Follow the platform-specific UDP port 53 and Docker DNS steps below.
• If the DNS server was reachable but rejected the query with NXDOMAIN or REFUSED, the resolver answered, so the UDP port 53 fix is not relevant. Check the resolver used by Docker, such as dnsmasq, Pi-hole, unbound, or systemd-resolved, and remove any forwarding rule, blocklist entry, or ACL that rejects registry.npmjs.org. If needed, configure Docker to use an organization-approved resolver that can resolve public names, restart Docker, and retry onboarding.

For an unreachable resolver, pick the matching platform path below, apply it, then re-run $$nemoclaw onboard.

• Linux with systemd-resolved. Add a DNSStubListenerExtra drop-in pointing at the docker bridge gateway IP (the preflight prints the detected IP), then add the same IP to /etc/docker/daemon.json under dns. Restart systemd-resolved and docker.
• macOS with Colima. Restart Colima with the corporate DNS address, for example colima stop && colima start --dns <corp-dns-ip>.
• macOS with Docker Desktop. Add the corporate DNS address to ~/.docker/daemon.json under dns, then restart Docker Desktop.
• Windows or WSL. Configure DNS in the Docker Desktop settings GUI, or apply the Linux fix above when running native docker inside WSL.

Verify the fix worked:

docker run --rm busybox nslookup registry.npmjs.org

When the lookup returns an answer, retry onboarding.

Direct DNS lookups fail in a Docker-driver GPU sandbox

Egress covered by OpenShell network policies resolves destinations through the gateway. A direct DNS lookup inside the agent network namespace can fail on Docker-driver GPU hosts such as DGX Spark even when policy-covered inference, messaging, and search work normally. Direct in-sandbox DNS depends on Docker and the host resolver and is not a supported NemoClaw network-policy path, so NemoClaw does not use it as a sandbox health check.

Run a manual lookup only when you are diagnosing a custom tool that performs its own DNS resolution:

openshell sandbox exec --name <sandbox-name> -- getent hosts host.openshell.internal
openshell sandbox exec --name <sandbox-name> -- getent hosts example.com

If the host alias resolves but the external name does not, Docker's embedded resolver may be forwarding to an upstream DNS server that the sandbox bridge cannot use. Do not treat this result as evidence that a policy-covered feature is unhealthy. Test the affected inference, messaging, or search request through its normal policy path and inspect denied requests with openshell term.

If a custom tool requires direct DNS, configure the Docker daemon to use a resolver that containers can reach. On hosts with VPN or split-DNS software, use an upstream resolver that remains reachable from the Docker bridge, then recreate or rebuild the sandbox. Keep bridge networking enabled so the sandbox retains its normal Docker network isolation.

Host DNS resolution is blocked before provider validation

NemoClaw also checks that the host process can resolve the provider host before it starts NVIDIA provider validation. A firewall rule that blocks host DNS traffic on port 53 can make later validation fail with curl: (6) Could not resolve host: integrate.api.nvidia.com even when container DNS probes look healthy. Current onboarding stops earlier with a host DNS diagnostic and remediation hints.

Verify host DNS outside NemoClaw:

node -e 'require("node:dns").resolve4("integrate.api.nvidia.com", (err, addrs) => { if (err) { console.error(err); process.exit(1); } console.log(addrs.join(",")); })'

Fix the host firewall, VPN, or DNS policy so the host can resolve the provider endpoint, then rerun onboarding. If you intentionally use a non-NVIDIA provider and need to bypass only this preflight, set NEMOCLAW_SKIP_HOST_DNS_PREFLIGHT=1.

Port already in use

The NemoClaw dashboard uses port 18789 by default and the gateway uses port 8080. If another sandbox already owns the dashboard port, onboarding scans ports 18789 through 18799 and uses the next free port. If all ports in that range are occupied, the error lists the owner for each port and suggests using --control-ui-port with a port outside the range.

NemoClaw allocates each Hermes sandbox an OpenAI-compatible API port from 8642 through 8652, so it rejects every port in that range as a dashboard port for any agent. When all ports in the API range are occupied, the error lists the owner for each port. Destroy a listed Hermes sandbox or stop a listed non-OpenShell listener, then rerun onboarding.

On macOS, the port check also tries a privileged lsof probe without prompting for a password so root-owned listeners are detected before the sandbox build starts. For a new sandbox, NemoClaw reserves the selected loopback port through sandbox preparation and the image build. If another listener claims the port before NemoClaw binds the reservation, NemoClaw selects another port before changing sandbox resources. NemoClaw releases the reservation immediately before openshell forward start runs. If forwarding then fails, onboarding removes the new sandbox and tells you to resolve the reported error before retrying.

When a previous onboard, upgrade, or sandbox crash leaves a stale openclaw-gateway host process holding the dashboard port, $$nemoclaw onboard --fresh, $$nemoclaw <name> destroy (when destroying the last sandbox), and $$nemoclaw uninstall automatically sweep the dashboard port range and signal SIGTERM then SIGKILL to recover. The sweep only targets processes owned by the current user whose command line matches openclaw-gateway or openshell forward markers, and skips dashboard ports owned by other live sandboxes.

If onboarding preflight resolves the complete listener set for a gateway port conflict, the diagnostic lists every listener. Each entry contains the process name and PID, or only the PID when NemoClaw cannot read the process name. The stop command targets only listeners that fail ownership verification. It does not target a verified managed listener. If NemoClaw resolves no listener, the diagnostic provides an lsof inspection command. Before you run a stop command, confirm that each targeted listener is not part of a second NemoClaw gateway environment. Release that environment with NEMOCLAW_GATEWAY_PORT=<port> $$nemoclaw uninstall instead of stopping its process.

If a non-NemoClaw process is already bound to the dashboard port or the gateway port, identify the conflicting process, verify it is safe to stop, and terminate it:

sudo lsof -i :18789
kill <PID>

If the process does not exit, use kill -9 <PID> to force-terminate it. Then retry onboarding.

Alternatively, override the conflicting port instead of stopping the other process. Pass --control-ui-port with the desired dashboard port:

$$nemoclaw onboard --control-ui-port 19000

You can also set CHAT_UI_URL with the desired port:

CHAT_UI_URL=http://127.0.0.1:19000 $$nemoclaw onboard

Or set the port directly:

NEMOCLAW_DASHBOARD_PORT=19000 $$nemoclaw onboard

For an OpenShell gateway port conflict, set NEMOCLAW_GATEWAY_PORT to a free non-privileged port that does not overlap NemoClaw's dashboard, vLLM, Ollama, or Ollama proxy ports:

NEMOCLAW_GATEWAY_PORT=8990 $$nemoclaw onboard

Remote/headless hosts should keep the OpenShell gateway on loopback and bind the dashboard forward instead:

NEMOCLAW_DASHBOARD_BIND=0.0.0.0 NEMOCLAW_GATEWAY_PORT=8990 $$nemoclaw onboard

Use NEMOCLAW_DASHBOARD_BIND=0.0.0.0 again on later $$nemoclaw <sandbox> connect calls. If the sandbox was originally created without remote bind, recreate it with the same onboard command plus --recreate-sandbox before connecting remotely.

NemoClaw rejects NEMOCLAW_GATEWAY_BIND_ADDRESS=0.0.0.0 for Docker-driver gateways while gateway JWT auth is active.

Use NEMOCLAW_GATEWAY_BIND_ADDRESS=0.0.0.0 only on supported gateway modes and only when other hosts on the network should be able to reach the gateway.

Older-glibc gateway compatibility container

OpenShell 0.0.101 directly supports Linux hosts with glibc 2.39 or newer. On an older trusted host, NEMOCLAW_OPENSHELL_GATEWAY_CONTAINER_PATCH=1 explicitly opts into NemoClaw's compatibility container. Leave it unset on supported hosts.

The compatibility container uses host networking and mounts the host Docker socket read-only. A read-only socket mount still permits privileged Docker API operations and can control the host, so do not enable this mode on an untrusted or shared host. The gateway remains loopback-bound, and startup fails closed unless the configured Unix socket answers as a Docker daemon. See Gateway Compatibility Container for the container boundary and removal conditions.

Refer to Environment Variables for the full list of port overrides.

Running multiple sandboxes simultaneously

Each sandbox requires its own dashboard port. If you onboard a second sandbox without overriding the port, onboarding uses the next free port in the 18789 to 18799 range. onboard checks openshell forward list before starting a new forward, so a second onboard cannot silently take over the first sandbox's port.

Assign a distinct port only when you want a specific value:

$$nemoclaw onboard                                                   # first sandbox uses default 18789
$$nemoclaw onboard                                                   # second sandbox uses the next free port
$$nemoclaw onboard --control-ui-port 19000                          # explicit port override

Each sandbox then has its own SSH tunnel and its own dashboard URL:

http://localhost:18789   ← first sandbox
http://localhost:19000   ← second sandbox

You can verify which tunnel belongs to which sandbox with:

openshell forward list
$$nemoclaw list

$$nemoclaw list prints the recorded dashboard URL for each sandbox. These dashboard ports are separate from the gateway-wide inference route.

A Gateway Port Stays Bound After Uninstall or Re-Onboard

A gateway port that keeps listening after you uninstall, or that keeps serving after you re-onboard without NEMOCLAW_GATEWAY_PORT set, can belong to a second environment.

Onboarding under a non-default NEMOCLAW_GATEWAY_PORT registers the sandbox on gateway nemoclaw-<port> and stores its registry and state under ~/.nemoclaw/gateways/<port>/. Every later command reads the state root that its own NEMOCLAW_GATEWAY_PORT selects, so a command run without that variable operates on port 8080 and neither reuses nor stops the other environment. Clearing the variable does not move an existing sandbox back to the default port.

List the gateways the host still has:

openshell gateway list

If this happened after the deprecated global $$nemoclaw stop, read the command's final status line. Without a resolved sandbox name, the command releases a gateway only when a valid, explicitly set NEMOCLAW_GATEWAY_PORT selects it. Host services stopped; managed gateway not released. means the command stopped scoped host services but intentionally left the gateway listener running. After openshell gateway list confirms the exact target port, rerun the deprecated full stop with that scope only when you intend to release that gateway:

NEMOCLAW_GATEWAY_PORT=9000 $$nemoclaw stop

If NemoClaw reports that release was not confirmed, inspect the remaining listener and stop it only after you verify that it belongs to the selected gateway.

If the gateway name and its port-scoped state remain, treat it as a second environment and select that port for cleanup. If the gateway is absent but the port still listens, cleanup did not stop the listener; follow the process or service remediation printed by uninstall before you retry. If uninstall reported that it kept an openshell-gateway process owned by another user running, that process still holds the port. This can happen after uninstall exits successfully because NemoClaw does not treat another user's process as a cleanup failure. Ask that user to stop the process, or onboard under a different NEMOCLAW_GATEWAY_PORT.

Remove one environment by selecting its port:

NEMOCLAW_GATEWAY_PORT=9000 $$nemoclaw uninstall

Remove every gateway port in one run:

$$nemoclaw uninstall --all-gateway-ports

After you confirm uninstall and it exits with status 0, run openshell gateway list again. For a NemoClaw-managed gateway without --keep-openshell, the gateway name that uninstall removed must be absent. An externally supervised gateway or a run with --keep-openshell preserves the gateway process and its resources.

The same scoping applies to $$nemoclaw <name> stop. When stop reports that no valid gateway binding is registered for a sandbox, the sandbox can be registered under a different gateway port. Rerun stop with that NEMOCLAW_GATEWAY_PORT value set. If that does not find the sandbox, resolve the missing, invalid, or unreadable registry entry that the command reports.

Refer to Uninstall NemoClaw for the full sweep contract.

A shared inference route conflicts with another sandbox

Refer to Use Shared Gateway Routes for route time-sharing, provider-global compatibility, and status drift fields.

If inference set reports a valid shared-route conflict, align the named sandbox records or remove a sandbox you no longer need. If onboarding or connect reports a provider-global identity conflict, align the same-name provider's custom endpoint, API family, and credential environment-variable name across the named sandboxes, or remove a conflicting sandbox you no longer need.

If the error names incomplete legacy custom-route metadata, back up and remove the affected sandbox, then re-onboard it with an explicit custom endpoint and API family. For an OpenAI-compatible route, replace the example endpoint, model, and sandbox name in this recovery sequence:

$$nemoclaw legacy-sandbox destroy
NEMOCLAW_PROVIDER=custom \
  NEMOCLAW_ENDPOINT_URL=https://endpoint.example/v1 \
  NEMOCLAW_MODEL=your-model-id \
  NEMOCLAW_PREFERRED_API=openai-completions \
  $$nemoclaw onboard --name legacy-sandbox

If the error names an invalid gateway binding, restore the affected row's known-good gatewayName and gatewayPort metadata from a trusted backup; otherwise back up and remove the sandbox, then re-onboard it. Do not guess or copy a binding from another sandbox because lifecycle commands use it to select the gateway.

Onboarding

Cgroup v2 errors during onboard

Older NemoClaw releases relied on a Docker cgroup workaround on Ubuntu 24.04, DGX Spark, and WSL2. Current OpenShell releases handle that behavior themselves, so NemoClaw no longer requires a Spark-specific setup step.

If onboarding reports that Docker is missing or unreachable, fix Docker first and retry onboarding:

$$nemoclaw onboard

Podman is not a tested runtime. If onboarding or sandbox lifecycle fails, switch to a tested runtime (Docker Desktop, Colima, or Docker Engine) and rerun onboarding.

Cluster fails with overlayfs snapshotter cannot be enabled on Docker 26+

Docker Engine 26 and later default fresh installations to the containerd image store, which exposes its layers via the overlayfs snapshotter rather than the legacy overlay2 graph driver. The k3s server inside the OpenShell cluster image needs to mount its own overlay filesystem on top, and the kernel rejects nesting two non-trivial overlay mounts. The cluster container then loops with:

"overlayfs" snapshotter cannot be enabled for "/var/lib/rancher/k3s/agent/containerd",
try using "fuse-overlayfs" or "native":
failed to mount overlay: ... err: invalid argument

This is a Docker default-driver change, not a NemoClaw or OpenShell regression. The same hardware uses the legacy overlay2 driver and is unaffected when it runs Docker 25 or earlier, or any Docker version with the containerd image store disabled.

NemoClaw detects the Docker 26+ containerd-snapshotter overlayfs configuration during onboarding and transparently builds a small drop-in replacement for the cluster image on the local Docker engine. The patched image installs fuse-overlayfs and selects it as the k3s snapshotter, bypassing the kernel-level nested-overlay limitation. No host configuration changes, sudo, or Docker restart required.

The auto-fix runs once per OpenShell version on the affected host. Subsequent onboarding runs reuse the cached patched image. Hosts without the conflict (Driver: overlay2 in docker info, macOS Docker Desktop, or Linux installations that disable the containerd image store) see no change in behavior.

Override knobs:

• NEMOCLAW_DISABLE_OVERLAY_FIX=1: skip the auto-fix and run against the unmodified upstream cluster image. Useful for diagnosis or when you have already applied the manual workaround below.
• NEMOCLAW_OVERLAY_SNAPSHOTTER=native: build the patched image with k3s's native snapshotter instead of fuse-overlayfs. The native snapshotter copies image layers instead of overlaying them, so it uses more disk but does not depend on FUSE. Default is fuse-overlayfs.

If you prefer to disable the new Docker storage driver instead of running the patched image, edit /etc/docker/daemon.json:

{
  "storage-driver": "overlay2",
  "features": { "containerd-snapshotter": false }
}

Then restart Docker (sudo systemctl restart docker) and re-run $$nemoclaw onboard. This restores the legacy overlay2 driver host-wide, which kills any other running containers. Prefer the auto-fix unless you need the change for unrelated reasons. Switching storage drivers also rebuilds the entire local image graph: previously-pulled images become unusable and Docker re-pulls them on first reference, so expect a cold cache and additional disk usage right after the restart.

OpenShell version above maximum

Each NemoClaw release validates against a range of tested OpenShell versions. If the installed OpenShell version exceeds the configured maximum, $$nemoclaw onboard exits with an error:

✗ openshell <version> is above the maximum supported by this NemoClaw release.
  blueprint.yaml max_openshell_version: <max>

Upgrade NemoClaw to a version that supports your OpenShell release, or install a supported OpenShell version from the OpenShell releases page.

For fresh installs, NemoClaw passes the blueprint range to install-openshell.sh and resolves a compatible published OpenShell release before downloading. If GitHub release metadata is unavailable, the script uses its bundled fallback pin and the post-install gate still enforces the configured range.

Installer Reports an OpenShell Gateway Version Mismatch

On Linux, an existing OpenShell package can provide a systemd user service that starts a different gateway version from the user-local version that NemoClaw installs. The installer stops before onboarding instead of using the two versions together. The error reports both gateway versions and binary paths.

Do not remove the existing OpenShell package if its gateway manages resources outside NemoClaw. Package removal can stop that gateway. Align the package with the version in the installer error, or plan the migration of those resources first.

If you no longer need the APT-installed OpenShell package, remove it and rerun the installer:

sudo apt remove openshell
curl -fsSL https://www.nvidia.com/nemoclaw.sh | bash

The next installer run must continue past the OpenShell installation step without reporting a version mismatch.

Docker Driver Gateway Reports a Missing Migration

Onboarding can stop when the Docker driver gateway log contains both parts of this error:

migration N was previously applied
is missing in the resolved migrations

NemoClaw identifies <selected-state-dir>/openshell.db as incompatible with the installed OpenShell migration set. This failure can happen after an OpenShell downgrade. Installing a NemoClaw release that is older than the installed one performs that downgrade, because each release pins one OpenShell version and the installer reinstalls OpenShell at the pin. You reach that state in one of three ways:

• You select an older release with NEMOCLAW_INSTALL_TAG or NEMOCLAW_INSTALL_REF.
• The default lkg release is older than the NemoClaw release already on the host.
• You install an older OpenShell yourself.

The diagnosis always prints the database path. When an unused archive path is available, it also prints that archive path beside the selected state directory and the profile-specific onboarding command. When no unused archive path is available, it asks you to keep the gateway stopped and inspect the state directory instead.

The selected state directory contains the gateway database, mutual TLS private keys, JSON Web Token signing material, and every sandbox and provider registration on the selected gateway. Moving it makes those registrations and credentials unavailable to the fresh gateway. Other sandboxes on the selected gateway can require re-onboarding and credential entry.

When a service manager owns the gateway, the printed recovery stops it in the same command chain that moves the state directory. NemoClaw cannot establish in advance that the directory stays free, because the managed service restarts the gateway on failure and a replacement can start at any point before the move. Running the stop inside the chain removes that gap.

NemoClaw resolves the owning service before it prints, so the stop names the unit that runs on this host: the upstream OpenShell package unit, the NemoClaw user service, or the Homebrew formula. When no service manager owns the gateway, NemoClaw runs it standalone. Before it offers the state move in that case, the gateway runtime checks the recorded process and scans current gateway process identities for the runtime namespace tied to the selected state directory. It withholds the move unless that scan establishes that the standalone gateway state is unused.

When NemoClaw prints the state move, run the exact commands it prints:

1. Stop the owning gateway service, when the printed chain includes that step.
2. Create the printed .incompatible archive with owner-only access. If that path exists, NemoClaw adds a numeric suffix instead of nesting or replacing an earlier archive.
3. Move the selected state directory into the archive as gateway-state.
4. Run the printed onboarding command only after the stop, the archive, and the move succeed. Standard onboarding prints $$nemoclaw onboard --resume. The portable experimental profile prints its required fresh-onboarding command for this gateway-state recovery.

The archive remains beside the selected state directory and retains the previous gateway records and credentials. Keep it owner-only until onboarding completes and every required sandbox and provider registration is restored. Delete the archive only after you no longer need its gateway records or credentials for recovery.

Installer Reports That the Systemd User Manager Is Unavailable

On Linux, an OpenShell package can install /usr/lib/systemd/user/openshell-gateway.service on a host without a reachable systemd user manager. The service query can then return this diagnostic:

Failed to connect to bus: No medium found

The installer accepts only recognized user-manager-unavailable diagnostics for the standalone gateway fallback. It checks .wants, .requires, and .upholds links in the standard systemd user unit paths. The installer keeps the standalone lifecycle only when neither gateway service has an activation path that can later claim port 8080. The installer also stops when SYSTEMD_UNIT_PATH overrides the standard paths. The installer does not parse, modify, or remove a package or foreign unit to make this decision.

If an activation path exists, the installer stops because the service can start later and compete for port 8080. Restore the systemd user manager, then inspect both possible services:

systemctl --user status openshell-gateway.service
systemctl --user is-enabled openshell-gateway.service
systemctl --user status nemoclaw-openshell-gateway.service
systemctl --user is-enabled nemoclaw-openshell-gateway.service

Resolve the competing service through its package or platform owner. Do not delete an activation link or edit a unit file by hand. Rerun the installer only after the owner confirms that no enabled user service can claim port 8080.

Unknown service query errors remain fatal. The installer also stops for malformed effective metadata, an untrusted unit or executable path, an executable failure, or a gateway version mismatch. Follow the reported condition instead of forcing the standalone fallback.

Sandbox build fails during OpenClaw plugin install

During sandbox creation, the OpenClaw image setup can install managed plugins for selected features such as web search or diagnostics. If the build reaches openclaw plugins install and the npm registry or ClawHub is blocked, NemoClaw classifies that narrow failure and prints a policy hint instead of only generic resume guidance. Brave Search uses an external OpenClaw plugin and can reach this install path. Tavily ships with the pinned OpenClaw runtime, so NemoClaw verifies the bundled extension instead of installing a separate Tavily package.

Check that the active policy and host network allow the npm registry and ClawHub endpoints needed by the plugin, or disable the feature that requested the plugin. For example, if the plugin is for web search, disable that feature and resume onboarding:

NEMOCLAW_WEB_SEARCH_PROVIDER=none $$nemoclaw onboard --resume

If you want the feature, fix the network or policy path first, then resume onboarding:

$$nemoclaw onboard --resume

Web search verification reports a warning or security error

When web search is enabled, onboarding checks the selected agent configuration and sends a real search request through the sandbox egress path. Configuration and egress verification are best effort, so those failed checks print a warning and let onboarding finish. The selected credential's live sandbox isolation check is required. If NemoClaw confirms that the raw Brave or Tavily key is visible, or the sandbox does not return a valid isolation result, it reports a security error. The CLI pauses onboarding and exits with a nonzero status. Recreate that sandbox through the supported onboarding flow before using it:

$$nemoclaw onboard --recreate-sandbox

First confirm that the provider credential and matching policy preset exist.

$$nemoclaw credentials list
$$nemoclaw <name> policy list

Look for <name>-brave-search with the brave preset or <name>-tavily-search with the tavily preset. Do not replace an openshell:resolve:env:<KEY> value in the sandbox configuration with a raw API key.

Confirm that OpenClaw reports the provider selected during onboarding.

$$nemoclaw <name> config get --key tools.web.search --format yaml

The provider should be brave or tavily and enabled should be true. If the provider is wrong, rerun onboarding with NEMOCLAW_WEB_SEARCH_PROVIDER=brave or tavily and the matching BRAVE_API_KEY or TAVILY_API_KEY.

Confirm that the generated Hermes configuration selects the Tavily backend.

$$nemoclaw <name> exec -- cat /sandbox/.hermes/config.yaml

The output should include a web mapping with backend: tavily. If it does not, rerun onboarding with NEMOCLAW_WEB_SEARCH_PROVIDER=tavily and TAVILY_API_KEY.

Rerunning onboarding with a different provider recreates the sandbox because the provider configuration and credential attachment are build-time inputs. NemoClaw validates the replacement key before it removes the existing sandbox, then backs up and restores the supported workspace state during recreation. If the configuration is correct but the egress probe fails, keep the matching preset applied and inspect the blocked request with openshell term before widening any policy rule.

Sandbox containers cannot reach the gateway

On native Linux Docker-driver hosts, $$nemoclaw onboard verifies the route that sandbox containers use to reach the OpenShell gateway. If a host firewall blocks that path, onboarding exits with output like:

✗ Sandbox containers cannot reach the gateway at host.openshell.internal:8080.
  A host firewall may be blocking traffic from the OpenShell Docker bridge.

Apply the ufw command printed by onboarding, then rerun onboarding. If the message does not include a subnet, derive it from the OpenShell Docker network:

SUBNET=$(docker network inspect openshell-docker --format '{{(index .IPAM.Config 0).Subnet}}')
sudo ufw allow from "$SUBNET" to any port 8080 proto tcp
$$nemoclaw onboard

This reachability check uses a disposable Docker probe and does not create or replace a sandbox. If Docker GPU compatibility recreation fails later, follow GPU routing or compatibility patch failed. That path can restore the pre-patch sandbox. If its diagnostics report manual cleanup, use only the printed exact-container command. That command targets the failed replacement and preserves the restored sandbox.

Custom OpenClaw image creates without a gateway or dashboard

$$nemoclaw onboard --from <Dockerfile> treats the supplied Dockerfile as the complete sandbox image rather than adding it on top of the stock managed runtime. If deployment verification cannot reach the gateway, NemoClaw checks for /tmp/gateway.log, /usr/local/bin/nemoclaw-start, and /sandbox/.openclaw/openclaw.json in the custom sandbox. When all three paths are absent, the CLI reports that the image lacks the NemoClaw-managed OpenClaw runtime and does not suggest repeated dashboard port-forward retries. This failure commonly occurs when the custom Dockerfile starts from ghcr.io/nvidia/nemoclaw/sandbox-base alone because that image is an intermediate dependency image.

Rebuild the custom image from the full stock Dockerfile and source context for the same NemoClaw release. For the version-pinned plugin workflow, refer to Install OpenClaw Plugins.

If the sandbox is unreachable or the managed runtime paths are present, NemoClaw retains the existing generic gateway-log and host OpenShell-log guidance because the base-only failure is not proven.

A custom image without the managed runtime can fail while NemoClaw starts the sandbox container. NemoClaw reports exit code 127 without assigning a cause unless captured logs contain the exact env error for missing nemoclaw-start. When that error is present, the failure output identifies the missing managed startup command and gives the same rebuild guidance. If NemoClaw saves pre-rollback diagnostics, the reported directory contains the captured container logs. If rollback succeeds, NemoClaw restores and starts the pre-patch sandbox container. It does not print a sandbox deletion command for the restored sandbox. If the failed replacement container remains, NemoClaw prints an exact-container Docker cleanup command. If NemoClaw cannot confirm whether the replacement remains but retains its validated exact ID, it prints the same target-safe command. Without a validated exact ID, it reports cleanup as unknown and prints no deletion command. If rollback fails, sandbox and container state can be uncertain. If NemoClaw reports a diagnostics directory, inspect it. Inspect the diagnostics before removing any container.

connect exits because the gateway is down

$$nemoclaw <name> connect checks the OpenShell gateway before it tries dashboard forwarding, SSH, or inference repair. If the gateway is not reachable, the command exits early and prints recovery guidance.

Resume onboarding so NemoClaw recreates or reconnects the managed gateway, then retry:

$$nemoclaw onboard --resume
$$nemoclaw <name> connect

Run $$nemoclaw status for a broader gateway health report.

If `nemohermes recover` reports that the Hermes secret-boundary validator is missing, the sandbox image predates the recovery-side validator that re-checks `/sandbox/.hermes/.env`. Current NemoClaw releases fail closed in this state: recovery reports that the validator is missing, leaves an otherwise healthy gateway untouched, refuses to claim the secret boundary was checked, and instructs you to re-image the sandbox with a current Hermes build.

Re-image the sandbox with a current Hermes build before retrying recovery:

nemohermes <sandbox-name> rebuild --yes
nemohermes <sandbox-name> recover

Sandbox container reports (unhealthy) while the agent gateway process is still alive

The in-sandbox OpenClaw gateway can drop its HTTP listener while its process stays alive. A restart-class configuration change makes the gateway restart itself in place, and if that restart fails the process parks with no listener (/tmp/gateway.log shows gateway startup failed: ... Process will stay alive). Docker then marks the container (unhealthy) even though pgrep still finds the gateway.

NemoClaw prevents restart-class configuration changes from causing this condition and recovers a gateway that stops serving:

• The generated sandbox config pins gateway.reload.mode to hot, so configuration changes never make the gateway restart itself out from under the sandbox supervisor.
• A serving watchdog inside the sandbox stops a gateway process that does not serve, and the supervisor relaunches it.

The watchdog treats the gateway as serving only while the local health endpoint answers 200 or 401, the same response requirement the sandbox applies when it waits for the gateway at startup. These probe outcomes count as not serving:

• The connection is refused.
• The probe times out.
• The connection is accepted and then closed without a response.
• The health endpoint returns an HTTP status other than 200 or 401.

The watchdog logs the cause of every not-serving probe. Its recovery bound depends on whether the gateway has ever served:

• After the gateway returns a serving response, the watchdog stops it on the fourth not-serving probe with no intervening serving response. At the default 30-second interval, this takes roughly two minutes.
• Before the gateway has ever served, the watchdog uses a longer boot grace window because it cannot distinguish a slow boot from a gateway that cannot serve. It stops the gateway on the 20th not-serving probe, roughly 10 minutes after launch at the default interval. Set NEMOCLAW_GATEWAY_WATCHDOG_BOOT_GRACE_PROBES to change this bound.

Each of these watchdog settings accepts a positive integer of up to nine digits:

• NEMOCLAW_GATEWAY_WATCHDOG_BOOT_GRACE_PROBES.
• NEMOCLAW_GATEWAY_WATCHDOG_INTERVAL_SECONDS.
• NEMOCLAW_GATEWAY_WATCHDOG_REFUSED_THRESHOLD.

The watchdog rejects any other value, logs which setting it rejected, and falls back to that setting's default.

Look for [gateway-watchdog] lines in $$nemoclaw <name> logs. Before recovery reaches its bound, each not-serving probe line shows the count and bound. Examples include (2/4 since the last serving response) and (7/20 since launch, having never served).

If the watchdog reports that this supervisor is no longer the gateway's parent, the process survived without the supervisor that relaunches it. Recover the sandbox with $$nemoclaw <name> recover.

If the watchdog reports health probe inconclusive, the probe itself could not run inside the sandbox. An inconclusive probe leaves the gateway untouched and preserves the current not-serving count, whether the watchdog is counting since launch or since the last serving response. If curl is missing, the watchdog reports this exact diagnostic:

[gateway-watchdog] curl is unavailable; serving watchdog disabled (#7377)

This diagnostic means that the watchdog is disabled. For a custom image, add curl to the image first. Then rebuild the sandbox:

$$nemoclaw <name> rebuild --yes

When the gateway is not serving, openclaw health can report gateway_transport_error, often 1006 abnormal closure (no close frame), while the container stays running with no restarts. An agent reply can come from the OpenClaw CLI's embedded in-process fallback instead of the gateway. Therefore, an agent reply does not confirm that the gateway is serving.

The watchdog converts a gateway that cannot serve into a process exit that the sandbox supervisor already knows how to relaunch. It does not correct the OpenClaw lifecycle condition that left the process running. It stays necessary until an OpenClaw gateway that cannot serve exits on its own.

Because of the hot pin, restart-class configuration changes made inside the sandbox log config reload requires gateway restart; hot mode ignoring and do not take effect until the gateway restarts. For example, openclaw plugins install logs that message until the gateway restarts. Apply them with a supervised restart:

$$nemoclaw <name> gateway restart

or rebuild the sandbox for changes that affect provisioning:

$$nemoclaw <name> rebuild --yes

Invalid sandbox name

Sandbox names must contain 1 to 19 characters. They must be lowercase, start with a letter, contain only letters, numbers, and single internal hyphens, and end with a letter or number. Consecutive hyphens (--) are not allowed. The CLI rejects names that do not match these rules. It prints a Try: <suggested-slug> recovery line whenever it can derive a valid lowercase, hyphen-separated form from the input, so passing --name MyAssistant reports Try: myassistant and you can rerun with the suggested slug.

The CLI writes the rejected value as a quoted preview instead of raw input. The preview reads at most the first 80 UTF-16 code units from the input and escapes each code unit outside printable ASCII as \uXXXX. Escaping can make the preview longer than 80 output characters. This prevents a rejected name from injecting control sequences into terminal or CI output.

Names that collide with global CLI commands are also rejected. Reserved names include onboard, list, deploy, setup, start, stop, status, debug, uninstall, credentials, and help. Using a reserved name would cause the CLI to route to the global command instead of the sandbox.

If the name does not match these rules or is reserved, the wizard exits with an error. Choose a name such as my-assistant or dev1.

Sandbox creation fails on DGX

On DGX machines, sandbox creation can fail if the gateway's DNS has not finished propagating or if a stale port forward from a previous onboard run is still active.

Run $$nemoclaw onboard to retry. The wizard cleans up stale port forwards and waits for gateway readiness automatically.

GPU Setup Fails with a Placeholder GPU Name

On Windows, WSL, and native Linux ARM64 hosts, some systems report a placeholder display adapter name even when no NVIDIA GPU firmware is present. This section also applies when preflight reports no GPU on an ARM64 Linux host whose nvidia-smi shows a non-placeholder GPU name. NVIDIA NIM and GPU-backed sandbox setup require a real NVIDIA GPU.

When the primary memory-query probe reports exactly one placeholder-named GPU row on a native or Docker Desktop-backed WSL ARM64 Linux host without firmware-confirmed NVIDIA platform metadata, onboarding runs one bounded Docker CUDA workload. When that probe reports a single non-placeholder NVIDIA GPU name on such a host and the NVIDIA kernel-driver interface (/proc/driver/nvidia) is absent, onboarding runs the same workload. A WSL2 host never provides that interface because Windows paravirtualizes the GPU through /dev/dxg. A non-placeholder name that does not identify an NVIDIA GPU or product family does not start the workload. NemoClaw treats a recognized NVIDIA product model from /sys/class/dmi/id/product_name or /sys/firmware/devicetree/base/model, or a known Tegra device node, as authoritative platform identity. Docker may pull the CUDA sample image from nvcr.io and keeps the image in the local cache after the container exits. The workload uses this command:

docker run --rm --gpus all nvcr.io/nvidia/k8s/cuda-sample@sha256:7c7540bdf1f942d4fb6db97069fd6c289471b54ac29e3c7fcdf914cf77af7d41

The run is bounded to 3 minutes. Set NEMOCLAW_WSL_GPU_PROOF_TIMEOUT_MS to a positive millisecond value to change that bound. Increase the value only when the image pull or GPU workload needs more than 3 minutes. A passing workload lets onboarding treat the detected GPU as eligible for GPU passthrough during that run. For Windows-on-Arm, this proof is a technical detection check and does not change the Unsupported product status or establish platform qualification. Refer to Platform Support and Launch Claims for the current support boundary. A failed or timed-out workload leaves the GPU unproven and does not enable GPU passthrough. The names-only unified-memory fallback does not run this workload and rejects denylisted names. WSL hosts that are not Docker Desktop-backed do not run the workload and continue to report the GPU as unavailable.

When GPU detection rejects the nvidia-smi report, preflight prints the failed check under the Local NIM unavailable — no GPU detected line, for example an absent /proc/driver/nvidia interface or a failed bounded CUDA proof. If NemoClaw rejects the detected GPU name during preflight, select a CPU or remote inference provider, or move the setup to a host with a supported NVIDIA GPU and current drivers.

Jetson/Tegra hosts support sandbox GPU passthrough through the compatibility route. Onboarding detects those hosts separately and propagates eligible host group IDs for selected /dev/nvmap, /dev/nvhost-*, and /dev/nvgpu/igpu0/* nodes plus real /dev/dri/renderD* character devices. If that path fails, follow the Jetson/Tegra compatibility guidance below instead of treating a missing nvidia-smi result as a placeholder adapter.

Colima socket not detected (macOS)

Newer Colima versions use the XDG base directory (~/.config/colima/default/docker.sock) instead of the legacy path (~/.colima/default/docker.sock). Some installations expose a top-level Colima socket at ~/.colima/docker.sock. NemoClaw checks all three paths. If neither is found, verify that Colima is running:

colima status

Sandbox build is slow or hangs (under-provisioned container runtime)

Default Colima ships with 2 vCPU and 2 GiB of memory, which is not enough headroom for the BuildKit-driven sandbox image build. On macOS Apple Silicon, the build can stall part-way through with no progress and no error, leaving the wizard waiting indefinitely.

Preflight inspects docker info for NCPU and MemTotal and prints a warning when the runtime falls below 4 vCPU or 8 GiB. In interactive onboarding, the warning prompt defaults to abort, so pressing Enter stops the run before the sandbox build reaches the likely stall point. Type y only when you intentionally want to continue on the smaller runtime. Non-interactive onboarding prints the warning and continues. On Colima, raise the resources before re-running onboard:

colima stop
colima start --cpu 6 --memory 12 --disk 100

On Docker Desktop, raise CPU and memory limits in Settings → Resources, then apply and restart.

To silence the warning when the host is intentionally small, set NEMOCLAW_IGNORE_RUNTIME_RESOURCES=1 before running $$nemoclaw onboard.

Managed Sandbox Image Build Requires Local BuildKit

On a local Docker-driver gateway, NemoClaw builds each generated OpenClaw or Hermes sandbox image with host-side BuildKit. The generated Dockerfiles include BuildKit-only file-mode, per-step network, and mount controls. NemoClaw stops before sandbox creation in these cases:

• The local build is disabled.
• The staged build context fails trust validation.
• Docker cannot start the build.
• The build exits with an error.

It does not send that generated Dockerfile to the OpenShell gateway's classic Docker API builder because that builder cannot enforce the same instructions.

Keep the local prebuild enabled, and verify that the host Docker installation provides BuildKit:

unset NEMOCLAW_SANDBOX_PREBUILD
docker info
docker buildx version

Repair Docker access or the Docker Buildx plugin when either Docker command fails. Then rerun the original onboarding or rebuild command. For a resumable onboarding session, run:

$$nemoclaw onboard --resume

A passing recovery completes the local BuildKit build before sandbox creation starts. If NemoClaw rejects the staged build context trust boundary, do not change its permissions or move its Dockerfile. Rerun the command so NemoClaw creates a new private staged context. If the new context is also rejected, preserve the complete error and stop instead of forcing the gateway builder.

This requirement does not change user-supplied --from contexts, which continue to use the OpenShell gateway builder. It also preserves the gateway fallback when a generated LangChain Deep Agents Code image does not complete its local prebuild.

Re-onboard fails because port 18789 is held by SSH

After destroying a sandbox and gateway, the SSH port-forward process for the dashboard can be left running. Re-running onboard then fails preflight with Port 18789 is not available. Blocked by: ssh.

Current NemoClaw detects this case and kills the orphaned SSH process automatically before retrying the port check. If you see the error on an older release, identify the SSH process and terminate it manually:

sudo lsof -i :18789
kill <PID>

Then re-run $$nemoclaw onboard.

Sandbox Keeps Using the Previous Messaging Credential

Rerunning $$nemoclaw onboard --non-interactive with a replacement TELEGRAM_BOT_TOKEN, DISCORD_BOT_TOKEN, SLACK_BOT_TOKEN, SLACK_APP_TOKEN, WECHAT_BOT_TOKEN, or MSTEAMS_APP_PASSWORD previously reported success while the sandbox kept using the old credential. When you rerun onboarding, NemoClaw evaluates every active messaging credential binding and compares each supplied credential with its SHA-256 hash in the sandbox registry. When you provide a replacement credential, NemoClaw runs the channel's configured checks before it backs up supported workspace and manifest-declared state, destroys the sandbox, recreates it, and restores the backup. Files outside those state paths are not preserved. If an available pre-recreation check fails, onboarding stops before it backs up supported workspace and manifest-declared state or destroys the existing sandbox. Discord and Microsoft Teams require non-empty replacement input but cannot prove upstream credential validity before recreation, so send a real test message after onboarding and confirm that the recreated sandbox receives it and responds. If the channel state changes during rotation, onboarding stops before it destroys the existing sandbox and asks you to retry with the updated state. If you do not supply a credential, or the supplied credential matches the recorded hash, the credential check does not trigger recreation. If you replace a credential for a channel that you stopped with channels stop, onboarding does not trigger recreation because that channel is inactive.

If you suspect a sandbox is still using a stale messaging credential, follow Rotate a Messaging Credential to export the replacement without putting it in shell history. Then rerun onboarding so the credential check runs:

$$nemoclaw onboard --name <sandbox> \
  --non-interactive --yes --yes-i-accept-third-party-software

Sandbox creation killed by OOM (exit 137)

On systems with 8 GB RAM or less and no swap configured, the sandbox image push can exhaust available memory and get killed by the Linux OOM killer (exit code 137).

NemoClaw automatically detects low memory during onboarding and prompts to create a 4 GB swap file. If this automatic step fails or you are using a custom setup flow, create swap manually before running $$nemoclaw onboard:

sudo dd if=/dev/zero of=/swapfile bs=1M count=4096 status=none
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile
echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab
$$nemoclaw onboard

Previous onboarding session failed

If a previous $$nemoclaw onboard attempt fails partway through (for example, a provider or inference-setup step reporting an error), NemoClaw records the failure in ~/.nemoclaw/onboard-session.json.

When you re-run the installer, it detects the failed session and does not silently retry it. Silent retry would loop on the same failure if your original choice, such as an unreachable provider, was the cause.

• In an interactive terminal, the installer prompts whether to resume the failed session or start fresh. Press R (or Enter) to retry the same session, or f to discard it and make fresh choices.
• In non-interactive mode (piped curl | bash with NEMOCLAW_NON_INTERACTIVE=1, CI, scripts), the installer refuses and exits with a non-zero status so a scripted re-run cannot loop. You must opt in to one of two paths explicitly:

Start over with new choices to discard the recorded session and provider/model selection.

curl -fsSL https://www.nvidia.com/nemoclaw.sh | bash -s -- --fresh

curl -fsSL https://www.nvidia.com/nemoclaw.sh | NEMOCLAW_AGENT=hermes bash -s -- --fresh

curl -fsSL https://www.nvidia.com/nemoclaw.sh | NEMOCLAW_AGENT=langchain-deepagents-code bash -s -- --fresh

Or use environment variables instead. Set them on the bash side of the pipe because only the right-hand process inherits them.

curl -fsSL https://www.nvidia.com/nemoclaw.sh | NEMOCLAW_FRESH=1 bash

curl -fsSL https://www.nvidia.com/nemoclaw.sh | NEMOCLAW_AGENT=hermes NEMOCLAW_FRESH=1 bash

curl -fsSL https://www.nvidia.com/nemoclaw.sh | NEMOCLAW_AGENT=langchain-deepagents-code NEMOCLAW_FRESH=1 bash

Retry the same session.

This is only useful if the original failure was transient, for example a network blip or a stopped Docker daemon, and not a wrong provider choice:

$$nemoclaw onboard --resume

For a checkpoint schema 4 portable session, the plain command restores the portable profile from the checkpoint. You can also state the matching profile explicitly:

$$nemoclaw onboard --experimental-profile portable --resume

Portable resume does not trust ambient Docker or Podman runtime selectors. It derives and verifies the recorded current-user rootless Podman authority before it continues onboarding. If NemoClaw reports unsafe ownership, type, or mode, correct that filesystem condition and retry. Portable onboarding always uses the .config directory beneath the canonical home reported by the operating system; changing HOME or XDG_CONFIG_HOME does not select another location. For a recorded alternate configuration root or other user ID, home, runtime root, endpoint, runtime kind, or ownership drift, do not edit the checkpoint; run fresh onboarding.

If NemoClaw reports that an active checkpoint uses schema 1, 2, or 3, the older checkpoint did not record enough profile and runtime authority for resume. NemoClaw preserves the session and exits before portable configuration, socket activation, or resource changes. Discard that active session and start fresh onboarding:

$$nemoclaw onboard --fresh

If you intend to use the portable experimental profile, select it again for fresh onboarding:

$$nemoclaw onboard --experimental-profile portable --fresh

OpenClaw resume does not repeat completed non-secret sandbox, web search, messaging, or resource choices. Resume also reuses registered web search and messaging credentials when the same onboarding session recorded their successful OpenShell registration and OpenShell still reports the exact expected name, type, and credential keys. If the session lacks that registration receipt, the provider is missing, or its binding does not match, interactive resume requests the credential again; non-interactive resume preserves the completed choice, reports the required environment variable, and exits so you can export it before retrying $$nemoclaw onboard --resume.

As a last resort, you can also delete the session file directly and re-run the installer:

rm ~/.nemoclaw/onboard-session.json

Kubernetes namespace not ready

If onboarding fails with Kubernetes namespace not ready, a previous failed or interrupted setup may have left stale OpenShell or NemoClaw state behind. Clean up the failed installation before re-running the installer:

$$nemoclaw uninstall --yes
curl -fsSL https://www.nvidia.com/nemoclaw.sh | bash

The normal uninstall path keeps user data under ~/.nemoclaw/, including sandbox registry metadata, backups, and saved credentials unless you explicitly remove them. If $$nemoclaw uninstall reports that the local uninstall script is missing, follow the CLI's security boundary: download the versioned NVIDIA/NemoClaw tag URL that it prints, inspect the script locally, run that local copy, and then retry the installer.

curl -fsSLo uninstall.sh <versioned-uninstall-url>
less uninstall.sh
bash uninstall.sh --yes
curl -fsSL https://www.nvidia.com/nemoclaw.sh | bash

Runtime

OpenShell gateway and OpenClaw gateway startup order

NemoClaw uses two gateway layers for OpenClaw sandboxes:

• The OpenShell gateway runs on the host side and owns sandbox lifecycle, provider routes, port forwards, and openshell sandbox list / status queries.
• The OpenClaw gateway runs inside the sandbox container and serves the OpenClaw dashboard, agent API, and sub-agent WebSocket traffic.

Start and recover them in this order: container runtime, OpenShell gateway, sandbox container, then the in-sandbox OpenClaw gateway.

Do not start the OpenClaw gateway by hand before the OpenShell gateway is healthy. NemoClaw cannot select, inspect, or reconnect the sandbox until OpenShell can see the owning gateway.

If the host rebooted or the OpenShell gateway is down, first run:

$$nemoclaw <name> status

The status command selects or starts the sandbox's recorded OpenShell gateway when possible, then checks whether OpenShell can still see the sandbox. If the sandbox container is present but stopped on a Docker-driver host, status can recover the labeled container and then re-query OpenShell. After the sandbox is visible again, use $$nemoclaw <name> recover only for the in-sandbox OpenClaw gateway and host forwards. Use $$nemoclaw <name> gateway restart when you intentionally need the in-sandbox gateway to reload supported runtime configuration.

Reconnect after a host reboot

After a host reboot, the container runtime, OpenShell gateway, and sandbox may not be running. Follow these steps to reconnect.

1. Start the container runtime.

   • Linux: start Docker if it is not already running (sudo systemctl start docker)
   • macOS: open Docker Desktop or start Colima (colima start)

2. Check the managed OpenShell gateway service.

   If a custom-port gateway is NemoClaw-managed, skip this service check and continue with the NemoClaw recovery step using the same environment value. Only the default port 8080 uses a NemoClaw-managed service. If the gateway is externally supervised, inspect and restart it through the supervisor declared by NEMOCLAW_GATEWAY_MANAGEMENT, regardless of port.

   On Apple Silicon macOS with Homebrew, let NemoClaw inspect and restart the official formula service. NemoClaw runs each Homebrew operation inside the checksum-verified temporary trust boundary. Continue to the NemoClaw recovery step below instead of running brew services directly.

   NemoClaw verifies the staged formula checksum and temporarily trusts only nvidia/openshell/openshell around each Homebrew inspection, start, or stop operation. If formula verification fails or Homebrew cannot grant or remove temporary trust, rerun the standard NemoClaw installer and then rerun onboarding:

   curl -fsSL https://www.nvidia.com/nemoclaw.sh | bash

   The standalone gateway is selected only when Homebrew or both the staged formula and installed keg are absent. When the formula or keg exists, Homebrew remains the lifecycle authority. NemoClaw does not switch to the standalone gateway after a Homebrew inspection, start, or stop failure. Follow the reported repair guidance instead of changing service ownership manually. If the installed service fails inspection, startup, or its health check, NemoClaw prints this log command:

   tail -n 200 "$(brew --prefix)/var/log/openshell/openshell-gateway.out.log" "$(brew --prefix)/var/log/openshell/openshell-gateway.err.log"

   Rerun the NemoClaw installer to restore the managed service for later onboarding runs.

   On Linux package installs, inspect and restart the upstream service.

   systemctl --user status openshell-gateway
   systemctl --user restart openshell-gateway

   If the service fails inspection, startup, or its health check, NemoClaw prints this log command:

   journalctl --user --unit openshell-gateway --no-pager --lines=200

   On Linux tarball installs, inspect and restart the marked NemoClaw service.

   systemctl --user status nemoclaw-openshell-gateway
   systemctl --user restart nemoclaw-openshell-gateway

   If the service fails inspection, startup, or its health check, NemoClaw prints this log command:

   journalctl --user --unit nemoclaw-openshell-gateway --no-pager --lines=200

   The tarball unit is under $XDG_CONFIG_HOME/systemd/user, or ~/.config/systemd/user when XDG_CONFIG_HOME is not absolute. It starts with your user session; NemoClaw does not enable lingering. On Linux, NemoClaw attempts the standalone fallback when a managed service fails inspection, startup, or its health check. The standalone gateway starts only after NemoClaw verifies exclusive ownership of the gateway port. The fallback does not bypass managed-service trust validation or unsafe environment configuration. These conditions remain hard failures:

   • Homebrew formula identity query, metadata, or official-tap validation errors
   • Foreign or symlinked systemd units, or an untrusted systemd executable identity
   • An invalid DOCKER_HOST or a symlinked service environment file

   Without Homebrew on macOS, or without a reachable systemd user manager on Linux, continue with the standalone recovery step below.

3. Check sandbox state.

   openshell sandbox list

   If the sandbox shows Ready, skip to step 5.

4. Recover the managed gateway (if needed).

   If the sandbox is not listed after the service restart, or systemctl --user is unavailable, first ask NemoClaw to reconnect through the recorded sandbox:

   $$nemoclaw <name> status

   If that cannot restore the gateway registration, resume onboarding to recreate the managed gateway metadata:

   $$nemoclaw onboard --resume

   Wait a few seconds, then re-check with openshell sandbox list. On Docker-driver hosts, NemoClaw also looks for OpenShell-labeled sandbox containers when the gateway is healthy but reports the sandbox as missing. It can start a stopped labeled container, or restore the latest GPU-backup sibling container name and start it.

5. Reconnect.

   $$nemoclaw <name> connect

   The gateway usually rotates its SSH host keys across a reboot. connect detects the resulting identity drift, prunes the stale openshell-* entries from ~/.ssh/known_hosts, and retries automatically. You do not need to edit known_hosts by hand or re-run $$nemoclaw onboard in this case.

1. Start host auxiliary services (if needed).

   If you use the cloudflared tunnel started by $$nemoclaw tunnel start, start it again:

   $$nemoclaw tunnel start

   OpenShell-managed channel messaging handles Telegram, Discord, Slack, WeChat, and WhatsApp at onboarding, not through a separate bridge process from $$nemoclaw tunnel start. WeChat and WhatsApp are experimental. To pause a single bridge without destroying the sandbox, use $$nemoclaw <name> channels stop <channel>.

If the sandbox remains missing after restarting the gateway, run $$nemoclaw <name> rebuild --yes while the local registry entry still exists. The rebuild path uses the recorded sandbox metadata and the snapshot flow to preserve supported workspace and agent state. If the sandbox was intentionally deleted and you want a clean setup instead, run $$nemoclaw <name> destroy to remove the stale local entry, then run $$nemoclaw onboard. Create a snapshot first when the sandbox is reachable enough to back up state. For details, refer to Create and Restore Snapshots.

Gateway Port Stays Bound After Destroying the Last Sandbox

Destroying the final sandbox with --cleanup-gateway stops the packaged OpenShell gateway service before it reaps host gateway processes, so the gateway port is released. The service is stopped, not disabled or removed, and the next onboarding run starts it again. If the service cannot be stopped, destroy exits non-zero and prints the status command for the service.

On Apple Silicon macOS with Homebrew, rerun the standard NemoClaw installer to restore the pinned formula and temporary trust contract:

curl -fsSL https://www.nvidia.com/nemoclaw.sh | bash

Then rerun $$nemoclaw <name> destroy --cleanup-gateway instead of stopping the Homebrew service directly.

On Linux, stop the service yourself, then rerun destroy. Use the service name that matches the install. For package installs:

systemctl --user stop openshell-gateway

For tarball installs:

systemctl --user stop nemoclaw-openshell-gateway

gateway restart or recover reports privileged control unavailable

Built-in OpenClaw and Hermes lifecycle commands require a running direct sandbox container that belongs to the named NemoClaw registry entry. The host uses registry-scoped privileged direct-container control to send an authenticated request to the controller for the live topology. It does not fall back to ordinary openshell sandbox exec, SSH, or a manual in-sandbox relaunch.

Current built-in images support two direct-container shapes. A direct root-entrypoint container uses the root PID 1 supervisor. An OpenShell-managed container uses /opt/openshell/bin/openshell-sandbox as PID 1, exactly one nonroot nemoclaw-start supervisor, and the installed root-owned mode 0500 managed controller. An arbitrary nonroot entrypoint that does not match that managed process shape fails with privileged control unavailable. Kubernetes and other deployments without a matching direct container also fail closed with privileged control unavailable. Run the lifecycle command from a supported direct-container deployment rather than trying to launch the gateway by hand.

On a direct-container deployment, first confirm that the sandbox is running:

$$nemoclaw <name> status

When recover repairs a stopped built-in gateway, NemoClaw repeats the recovery action only for these exact transient results:

• Status 1 with blank stdout and exactly one stderr line, SUPERVISOR_BUSY.
• Status 137 with blank stdout and stderr.
• Status 1 with blank stdout and exactly one stderr line, Error response from daemon: Container <id> is restarting, wait until the container is running.

For the Docker result, <id> must be a 64-character lowercase hexadecimal ID that matches the selected registry-owned container. Recovery makes at most 11 controller attempts in total. It stops after 3 of those attempts return SUPERVISOR_BUSY. Managed settle confirmation treats exact SUPERVISOR_BUSY as inconclusive within its configured window. Status 137 and the Docker restart result remain terminal during that confirmation. The managed supervisor startup waiter accepts the two container-transition results within its separate 11-attempt bound. Unbound container IDs, reformatted Docker errors, status 137 with nonblank output, and other diagnostic results stop immediately. NemoClaw treats SUPERVISOR_UNAVAILABLE as terminal because it can report unreadable or untrusted supervisor state, ambiguous discovery, or a process-identity change. SUPERVISOR_NOT_RUNNING is a separate result that requires two zero-supervisor scans with a stable PID 1 and does not enter that retry loop. On a supported local Docker-driver sandbox with the legacy keepalive startup, it can authorize a container-identity-pinned recreation that commits only after managed health and settle checks pass. To bypass that trusted recreation while troubleshooting, run NEMOCLAW_DISABLE_SUPERVISOR_RELAUNCH=1 $$nemoclaw <name> recover; NemoClaw leaves the container unchanged and returns rebuild or re-onboard guidance. If recovery stops after 3 SUPERVISOR_BUSY results, or if gateway restart reports SUPERVISOR_BUSY, wait for the active request to finish and retry the command. If recovery exhausts the transition bound after status 137 or the Docker restart result, wait for the container to finish restarting and retry the command. If the error mentions SUPERVISOR_NOT_RUNNING and trusted recreation could not proceed, SUPERVISOR_REBUILD_REQUIRED, a missing nemoclaw-gateway-control helper, or a missing managed controller, the sandbox image may predate the current lifecycle contract. An exact SUPERVISOR_UNAVAILABLE result instead means the managed controller refused the current supervisor state rather than guessing which same-UID process is the gateway. The current recovery action and any managed settle confirmation stop immediately. If recover reports this result, follow its host-side gateway restart guidance. If restart also reports SUPERVISOR_UNAVAILABLE, or the image is incompatible, rebuild the image:

$$nemoclaw <name> rebuild --yes

For a custom image, update its Dockerfile to preserve the current NemoClaw entrypoint, root-only gateway control helper, root-only managed controller, and shared supervisor library before rebuilding.

Hermes config or shields reports a mutation already in progress

Hermes host config writes, shields transitions, and lifecycle seals use the same root-only mutation lock. A host-side config write is bound to the SHA-256 digest of the matching read, installs fresh sealed config inodes, refreshes both config hashes, and restores the prior permissions before releasing that lock. For shields changes, the lock remains held through recursive state-directory updates, config verification, and content-seal capture. If a concurrent config or shields command reports Hermes config mutation is already in progress, do not remove /run/nemoclaw/hermes-config-mutation.lock manually. Wait for the active config, shields, recovery, or restart command to finish, then retry. If the command says shields are up, run $$nemoclaw <name> shields down before config set or inference set.

A Hermes command reports that runtime provider state mutation owns direct-container execution

A current managed Hermes image on the Docker driver can report this retryable refusal while an exact Shields provider fence is active:

Runtime provider state mutation owns direct-container execution for sandbox '<name>'; retry after the provider fence is released.

The refused command did not start a process inside the sandbox. Do not bypass the fence with manual docker exec, OpenShell, or SSH commands, and do not delete ~/.nemoclaw/state/runtime-provider-lifecycle/. Wait for the active Shields or recovery command to finish, then reconcile the retained authority from the host:

$$nemoclaw <name> shields status

Use the status result as follows:

• If status reports verified UP, DOWN, or mutable-default posture and exits with status 0, retry the original command.
• If status reports runtime-provider recovery restored lockdown, retry the intended shields up or shields down transition because provider verification rejected the persisted mutable posture.
• If status reports DOWN (DRIFTED...) or NOT CONFIGURED (DRIFTED...), run $$nemoclaw <name> shields up. Open a new shields-down window only after shields status reports UP and exits with status 0.

Hermes can also log [SECURITY] Hermes startup held by an active runtime state mutation. before it reads mutable state. Do not kill or manually restart the held entrypoint. Run $$nemoclaw <name> shields status from the host so NemoClaw can recover the exact retained target and authenticate the startup release. If recovery still fails, preserve the owner-only lifecycle ledger and the complete error for diagnosis instead of removing a marker or changing in-sandbox permissions.

Hermes startup reports HERMES_CONFIG_MUTATION_ORPHANED

This refusal means a root-owned config or shields transaction stopped without a safely recoverable complete state. NemoClaw intentionally does not guess whether a partially updated recursive state tree should be locked or mutable, and it does not treat a leftover lock file as permission to continue. Do not delete /run/nemoclaw/hermes-config-mutation.lock, the restart state, or the persistent transaction marker manually. The config root remains sealed from the sandbox identity, so an ordinary in-place rebuild cannot create a new trustworthy backup and will stop before deleting the sandbox. If you already have a trusted host-side snapshot, record its selector, destroy the sealed sandbox, re-onboard the same sandbox name from trusted host configuration, and then restore that snapshot:

$$nemoclaw <name> snapshot list
$$nemoclaw <name> destroy
$$nemoclaw onboard --name <name> --agent hermes
$$nemoclaw <name> snapshot restore <selector>

Destroying the sealed sandbox permanently discards any state newer than the selected snapshot, so verify that the host-side snapshot exists before confirming destruction. Without a trusted pre-incident snapshot, automatic state-preserving recovery is not available. Recreate the sandbox from host-side onboarding configuration only if you accept losing the inaccessible in-sandbox state.

Hermes startup reports HERMES_RESTART_SEAL_ORPHANED

This refusal means a container recreation discarded the root-only restart metadata under /run while the persistent /sandbox tree still carries the frozen transaction marker, or a non-root entrypoint found a root transaction it cannot safely restore. NemoClaw cannot safely infer the original ownership, modes, or inode flags from a partial seal, so startup fails closed even when config.yaml and .env still look readable. Do not repair the paths with manual chown, chmod, or hash regeneration. An ordinary rebuild cannot safely back up a sealed tree. Restore a trusted snapshot into a new sandbox as shown above, or recreate from host-side onboarding configuration if no state-preserving recovery is required.

Hermes startup reports HERMES_LOCKED_PARENT_UNPROTECTED

This refusal means Hermes config files look shields-locked but /sandbox is not root:sandbox 1775, so the sandbox identity could rename the entire .hermes lock root. shields up cannot repair this after PID 1 has refused startup. Do not change the parent ownership manually or accept the current bytes as a new seal. Restore a trusted pre-incident snapshot into a new sandbox, or recreate the sandbox from host-side onboarding configuration.

OpenClaw startup reports OPENCLAW_LOCKED_PARENT_UNPROTECTED

This refusal means the OpenClaw config directory is root-owned but /sandbox is not root:sandbox 1775, so the sandbox identity could rename the entire .openclaw lock root. PID 1 refuses startup before migration or config reads, and shields up cannot repair the parent after that refusal. Do not change the parent ownership manually or accept the current bytes as a new seal. If you have a trusted host-side snapshot from before the incident, record its selector, destroy the refused sandbox, re-onboard the same sandbox name from trusted host configuration, and restore that snapshot:

$$nemoclaw <name> snapshot list
$$nemoclaw <name> destroy
$$nemoclaw onboard --name <name> --agent openclaw
$$nemoclaw <name> snapshot restore <selector>

Destroying the refused sandbox permanently discards state newer than the selected snapshot, so confirm that the host-side snapshot exists before destruction. Without a trusted pre-incident snapshot, recreate the sandbox from host-side onboarding configuration only if you accept losing the inaccessible in-sandbox state.

Sandbox is running an outdated agent version

After upgrading NemoClaw, $$nemoclaw <name> connect and $$nemoclaw <name> status warn if the sandbox is running an older agent version than the current image.

To upgrade the sandbox while preserving workspace state, run:

$$nemoclaw <name> rebuild

The rebuild command backs up state, destroys the old sandbox, recreates it with the current image, and restores state. Create a snapshot before rebuilding if you want an additional safety net:

$$nemoclaw <name> snapshot create
$$nemoclaw <name> rebuild

Sandbox shows as stopped

When status reports sandbox_container_stopped, Docker still has a container for the sandbox, but the container is not running. Use the lightest recovery path first instead of rebuilding immediately.

1. Confirm Docker can still see the labeled container.

   docker ps -a --filter "label=openshell.ai/sandbox-name=<name>"

2. Run recovery from the host.

$$nemoclaw <name> recover

For a stopped, non-paused Docker-driver container, recover starts the existing container before it waits for OpenShell readiness. It leaves a running or paused container unchanged. If Docker cannot start the container, the command continues to the readiness check and reports the resulting failure.

$$nemoclaw <name> status

Deep Agents Code does not provide the recover command. Use $$nemoclaw <name> start to start its existing container.

1. Check status from the host.

   $$nemoclaw <name> status

   On Docker-driver hosts, status also attempts non-destructive recovery when OpenShell reports the sandbox as missing but Docker still has a stopped openshell.ai/sandbox-name=<name> container or the latest GPU-backup sibling. A successful recovery prints that the sandbox was recovered from Docker and then shows the refreshed OpenShell state.

2. Rebuild only if the sandbox cannot be restarted or status still cannot recover it while the local registry entry exists:

   $$nemoclaw <name> rebuild --yes

   Rebuild recreates the sandbox from recorded metadata and preserves supported workspace and agent state. If the sandbox was intentionally deleted and you want a clean setup, run $$nemoclaw <name> destroy to remove the stale local entry, then run $$nemoclaw onboard.

Sandbox is registered locally but missing from the gateway

After a gateway restart, host reboot, or manual OpenShell cleanup, NemoClaw may still have a local registry entry for a sandbox that the live gateway no longer lists. $$nemoclaw <name> status and $$nemoclaw <name> connect preserve that local registry entry and print recovery guidance instead of deleting it automatically. Run $$nemoclaw <name> rebuild --yes when you want NemoClaw to recreate the sandbox from the recorded metadata, or run $$nemoclaw <name> destroy when you intentionally want to remove the stale entry.

A command reports that the registry file is not valid JSON

Registry operations that require complete sandbox records, such as $$nemoclaw list and $$nemoclaw onboard, stop with Configuration file is present but is not valid JSON, followed by the path to sandboxes.json and the recovery commands.

These operations stop instead of reading the file as an empty registry, so they cannot replace your sandbox records with empty state. Optional messaging health checks omit registry-derived information when they cannot read the registry. It does not rename, move, or rewrite the file.

Follow Malformed Registry File to keep a copy and remove it.

Status shows "not running" inside the sandbox

This is expected behavior. When checking status inside an active sandbox, host-side sandbox state and inference configuration are not inspectable. The status command detects the sandbox context and reports "active (inside sandbox)" instead.

Run openshell sandbox list on the host to check the underlying sandbox state.

Deep Agents

Deep Agents Config Lock Failure Recovery

A CRITICAL Deep Agents config-lock diagnostic can report fail-closed containment=, rollback failed, or that the lock rollback could not restore the trusted posture. A containment result identifies one of two confirmed postures or an incomplete containment attempt. A rollback failed result or lock-rollback diagnostic does not confirm containment. Both rollback diagnostics mean NemoClaw could not restore or confirm the original trusted posture.

• Config-root posture (fail-closed containment=config-root) means NemoClaw installed fresh 0444 root:root config and hash inodes. NemoClaw also confirmed 0500 root:root on /sandbox/.deepagents and 1775 root:sandbox on /sandbox.
• Sandbox-parent posture (fail-closed containment=sandbox-parent) means NemoClaw confirmed 0700 root:root on /sandbox. NemoClaw uses this posture when it cannot confirm the complete config-root posture.
• fail-closed containment=incomplete means NemoClaw could not confirm either complete posture.

Preserve the complete CRITICAL diagnostic. Do not retry shields up. Do not run chmod, chown, or another repair inside the sandbox. A confirmed containment posture removes the sandbox identity's access to the Deep Agents configuration. An incomplete containment result, a rollback failed result, or a lock-rollback diagnostic does not establish a trustworthy boundary from which to accept the current bytes. An ordinary rebuild cannot turn the current state into a trustworthy snapshot.

If you have a trusted host-side snapshot from before the failure, list the snapshots and record its selector:

$$nemoclaw <name> snapshot list

Destroying the sandbox permanently discards state newer than the selected snapshot. Confirm that the trusted host-side snapshot exists before you destroy the sandbox.

Destroy the sandbox, re-onboard the same name from trusted host configuration, and restore the snapshot:

$$nemoclaw <name> destroy
$$nemoclaw onboard --name <name> --agent dcode
$$nemoclaw <name> snapshot restore <selector>

For snapshot contents and selector rules, refer to Create and Restore Snapshots.

If no trusted snapshot exists and you do not need to preserve the current state, recreate the sandbox from host-side onboarding configuration.

This recreation permanently discards the current sandbox state. Continue only if you accept that loss.

$$nemoclaw <name> destroy
$$nemoclaw onboard --name <name> --agent dcode

After either recovery path, verify the recreated sandbox from the host:

$$nemoclaw <name> status
$$nemoclaw <name> shields status

Continue only when status identifies the expected Deep Agents sandbox and shields status returns without a CRITICAL or corrupt-state diagnostic. Then retry the original shields up operation.

dcode status reports a stale inference route

The managed dcode runtime reads provider and model settings from /sandbox/.deepagents/config.toml. NemoClaw generates that file during onboarding and rebuilds, and Deep Agents provider or model changes require fresh named recreation rather than inference set.

Check the host-recorded route first:

nemo-deepagents <name> status

Then connect and compare the in-sandbox identity:

nemo-deepagents <name> connect
dcode status

If the provider or model does not match the host route, rebuild the sandbox to regenerate config.toml from recorded metadata:

nemo-deepagents <name> rebuild

To intentionally switch provider or model, recreate the named sandbox with fresh onboarding:

nemo-deepagents onboard --fresh --name <name> --recreate-sandbox

Trusted route-probe helper is missing

Deep Agents sandboxes created by older NemoClaw images may not contain the image-owned /usr/local/lib/nemoclaw/dcode-managed-exec helper. Current connect, status, and doctor fail closed when that helper is missing because NemoClaw cannot run the authoritative inference.local route probe safely.

If the output says the trusted Deep Agents Code route-probe helper is missing, rebuild the sandbox with the current image and retry the command:

nemo-deepagents <name> rebuild
nemo-deepagents <name> status

dcode refuses to start because upstream auth state exists

The managed launchers refuse upstream credential state inside /sandbox/.deepagents/.state/auth.json and /sandbox/.deepagents/.state/chatgpt-auth.json. Those files can contain provider credentials or OAuth state that bypasses NemoClaw's host-owned credential boundary.

Remove the upstream auth state from the sandbox, then start dcode again:

rm -f /sandbox/.deepagents/.state/auth.json /sandbox/.deepagents/.state/chatgpt-auth.json
dcode status

Do not put provider credentials in /sandbox/.deepagents/.env, project .env files, or Deep Agents config files. Register credentials with NemoClaw or OpenShell on the host so the gateway can inject them at egress.

Managed MCP commands report an older Deep Agents runtime

For the current rebuild and retry workflow, refer to Agent MCP Capability Is Missing.

Tavily remains blocked after opt-in

Deep Agents does not have a NemoClaw-managed web-search feature. The Tavily flow only opens Python egress for project code or manually configured tools that call Tavily.

Confirm that the target sandbox has the tavily preset applied:

nemo-deepagents <name> policy list

If it is missing, apply the preset, register the host-side credential, and rebuild so the provider attaches:

nemo-deepagents <name> policy add tavily --yes
export TAVILY_API_KEY=tvly-...
nemo-deepagents credentials add tavily-search --type tavily --credential TAVILY_API_KEY
unset TAVILY_API_KEY
nemo-deepagents <name> rebuild

If Tavily is still blocked after rebuild, inspect recent policy denials:

nemo-deepagents <name> logs --tail 50

The tavily preset is a managed-Python opt-in. It is process-wide for sandbox Python and is not a dcode-only boundary.

Deep Agents read-only path checks fail

The Deep Agents image keeps the managed Python environment under /opt/venv read-only and leaves /sandbox writable for project and agent state. Writes under /usr, /etc, or /opt/venv should fail, while writes under /sandbox and /tmp should work.

If a read-only path probe reports that protected paths are writable, rebuild with the current NemoClaw image:

nemo-deepagents <name> rebuild

If startup reports that Landlock enforcement is unavailable, the Deep Agents sandbox fails closed instead of running with reduced filesystem enforcement. Deep Agents uses compatibility: strict for its managed filesystem policy, so kernels older than 5.13 or VM-backed Docker runtimes without Landlock support can block sandbox creation. Move the sandbox to a Linux kernel and container runtime that support Landlock, then rerun onboarding or rebuild the sandbox.

Git clone fails with a certificate verification error

In networks that inspect TLS, OpenShell injects a proxy CA bundle into the sandbox. Current NemoClaw exports that bundle as GIT_SSL_CAINFO during sandbox startup and persists it for $$nemoclaw <name> connect sessions, so Git can trust the proxy CA. It also forwards standard CA bundle variables for subprocesses, including GIT_SSL_CAPATH, CURL_CA_BUNDLE, and REQUESTS_CA_BUNDLE.

If Git still reports server certificate verification failed, reconnect to the sandbox and check that the CA variables are present:

env | grep -E 'SSL_CERT_FILE|GIT_SSL_CAINFO|CURL_CA_BUNDLE|REQUESTS_CA_BUNDLE' || true

grep exits non-zero when it finds no matches, so empty output (with the trailing || true) simply means none of these CA variables are set in the current shell.

If they are missing on an older sandbox, upgrade NemoClaw and run:

$$nemoclaw <name> rebuild

External channel TLS fails behind a corporate MITM proxy (NET:FAIL)

On networks where a corporate proxy re-signs external TLS, endpoints such as api.telegram.org can fail certificate verification even when network policy allows the connection. Logs show the request opening as NET:OPEN ... api.telegram.org:443 followed by NET:FAIL because the corporate root is not in the OpenShell trust path.

Provide the corporate CA before onboarding, then onboard or rebuild the sandbox.

export NEMOCLAW_CORPORATE_CA_BUNDLE=/path/to/corporate-ca.pem
$$nemoclaw onboard

Refer to Configure Corporate CA Trust for source precedence, host anchor discovery, image and runtime trust, custom Dockerfile requirements, import validation, and NEMOCLAW_CORPORATE_CA_IMPORT=0. If the import does not occur, check onboarding output for baking corporate proxy CA from ... or a warning that the selected source was skipped.

A request inside the sandbox fails with CONNECT tunnel failed, response 403

Sandbox outbound network access is denied by default and enforced by the OpenShell proxy. When a request targets a host that no applied policy preset allows, the proxy refuses the tunnel and tools surface only the protocol-level error:

fatal: unable to access 'https://example.com/foo/bar/': CONNECT tunnel failed, response 403
curl: (56) CONNECT tunnel failed, response 403

This is a network-policy denial, not a tool or certificate problem.

When you run a command through $$nemoclaw <name> exec -- ... and it exits non-zero, NemoClaw checks the sandbox audit log for a policy denial recorded after the command started. If it finds one, it appends a short breadcrumb to stderr after the tool's own output, naming the denied host:port when it can be extracted safely and showing the commands below:

curl: (56) CONNECT tunnel failed, response 403
$$nemoclaw: recent network policy denial detected for example.com:443 inside sandbox 'oc-fresh'.
  The sandbox's egress policy blocked this request; the tool above only saw the proxy's 403.
  See the denied flow:    $$nemoclaw oc-fresh logs --tail 50
  Review applied presets: $$nemoclaw oc-fresh policy list
  Allow the host:         $$nemoclaw oc-fresh policy add <preset>
  Silence this hint:      export NEMOCLAW_NO_POLICY_HINT=1

If the log probe fails, no breadcrumb is added. Unsafe endpoint data is omitted from the breadcrumb, and an invalid sandbox name is shown as <name>.

An IPv6 target is named in its RFC 3986 bracketed form:

$$nemoclaw: recent network policy denial detected for [2001:db8::1]:443 inside sandbox 'oc-fresh'.

The tool's own stdout/stderr bytes and its exit code are left unchanged. The breadcrumb is printed by the host CLI after the command finishes, and only for a genuine failure with a fresh denial. A command that succeeds, or one that fails for an unrelated reason, prints no breadcrumb. Set NEMOCLAW_NO_POLICY_HINT to any non-empty value other than 0 or case-insensitive false (for example, 1, true, TRUE, yes, or YES) to suppress it entirely.

The first interactive $$nemoclaw <name> connect shell also prints a one-line reminder of this denial signature and the logs command below. The reminder is shown once per top-level interactive session, and only when all of these hold: an egress proxy is configured, the shell is interactive with a terminal attached to stderr, and it is a top-level shell (not a nested subshell or pane). Suppress it with NEMOCLAW_NO_POLICY_HINT=1. The reminder names the sandbox when NemoClaw receives a valid sandbox name during sandbox creation. If no valid name is available, it shows <name>; run $$nemoclaw list to see your sandbox names. If the reported sandbox name contains characters that are not valid in a sandbox name (uppercase letters, underscores, control characters, and similar) or exceeds 19 characters, the reminder shows the <name> placeholder for safety rather than echoing the untrusted value. The reminder is intentionally proactive: the denial itself is surfaced by the OpenShell proxy, so the curl/git error text is left unchanged and the reminder points you to the logs instead.

To see which rule denied the request, read the merged logs from the host:

$$nemoclaw <name> logs --tail 50

If the host should be reachable, allow it with a preset or a [custom preset](../network-policy/customize-network-policy#custom-preset-files): If the host should be reachable, allow it with a built-in preset or apply a [reviewed custom preset file](../network-policy/configure-policies/create-custom-policy-presets) from the host:

$$nemoclaw <name> policy add <preset>

nemo-deepagents <name> policy add --from-file ./my-preset.yaml --yes

Replace <preset> with a real preset name such as github, pypi, or npm. Run $$nemoclaw <name> policy add with no preset to list the available presets.

Sandbox creation reports a TLS certificate mismatch

If sandbox creation reports a TLS or certificate mismatch, the OpenShell gateway certificate may have changed since the CLI last registered it. Remove the stale local gateway registration and then resume onboarding so NemoClaw refreshes the registration:

openshell gateway remove nemoclaw
$$nemoclaw onboard --resume

openclaw update hangs or times out inside the sandbox

This is expected for the current NemoClaw deployment model. NemoClaw installs openclaw into the sandbox image at build time, so the CLI is image-pinned rather than updated in place inside a running sandbox.

Do not run openclaw update inside the sandbox. Instead:

1. Upgrade to a NemoClaw release that includes the newer openclaw version.
2. If you build NemoClaw from source, bump the pinned openclaw version in Dockerfile.base and rebuild the sandbox base image.
3. Run $$nemoclaw <name> rebuild to recreate the sandbox with the updated image. The rebuild command automatically backs up workspace state before destroying the old sandbox and restores it afterward.

AWS EC2 Instance-Role Credential Discovery Is Unavailable

This is expected in an OpenClaw sandbox. NemoClaw forces AWS_EC2_METADATA_DISABLED=true because OpenShell blocks the link-local EC2 Instance Metadata Service endpoint. Existing sandboxes must use a current NemoClaw image before they receive this environment invariant.

Upgrade NemoClaw and rebuild the sandbox:

$$nemoclaw update --yes
$$nemoclaw <name> rebuild --yes

Reconnect and verify the value:

$$nemoclaw <name> connect
printenv AWS_EC2_METADATA_DISABLED

Expected output:

true

Only EC2 instance-role discovery is disabled. Static access keys, bearer tokens, shared profiles, SSO and process credentials, web identity, and ECS container credentials remain eligible. NemoClaw's host-local Amazon Bedrock adapter is outside the sandbox credential-discovery boundary and remains available. Do not add 169.254.169.254 to a network policy or override the variable.

Inference requests time out

Verify that the inference provider endpoint is reachable from the host. Check the active provider and endpoint:

$$nemoclaw <name> status

The main Inference line probes https://inference.local/v1/models from inside the sandbox and then sends one inference request over the same route, so it reflects the route the agent uses. If that line shows unauthorized, unhealthy, unreachable, or not probed, inspect the labeled diagnostic lines to identify the failing hop. An unauthorized line means the route answered but rejected the request, so refresh the provider credential rather than the route. For local Ollama and local vLLM, Inference (ollama backend) or the corresponding local-backend line reports the host-side service separately. For Local Ollama, current releases can also print Inference (auth proxy) when a proxy token is available. If a local backend or auth-proxy diagnostic fails, start the backend or re-run onboarding so NemoClaw can recreate the proxy token, restart the proxy, and refresh the route. For Ollama-backed OpenClaw sandboxes, agent passthrough uses the registered host route to warm an unloaded model after an Ollama daemon restart. If that bounded warm-up fails or times out, NemoClaw reports the result and continues so OpenClaw can emit its canonical backend error.

If the endpoint is correct but requests still fail, check for network policy rules that may block the connection. Then verify the credential and base URL for the provider you selected during onboarding.

If you entered an AWS Bedrock Runtime URL such as https://bedrock-runtime.us-east-1.amazonaws.com in the Other Anthropic-compatible endpoint flow, NemoClaw auto-detects it and routes sandbox traffic through a host-local adapter. Use the raw Bedrock Runtime host, not an Anthropic /v1/messages path, and verify that the model ID or inference profile ID is valid for that region. For auth, export AWS_BEARER_TOKEN_BEDROCK, AWS_PROFILE, or standard IAM environment credentials before onboarding; if you paste a key at the COMPATIBLE_ANTHROPIC_API_KEY prompt, NemoClaw uses it only as the adapter's Bedrock bearer token. Region errors usually mean the pasted endpoint region, AWS_REGION, AWS_DEFAULT_REGION, or the model/inference profile ID do not match.

For Ollama, vLLM, NIM, and compatible-endpoint inference validation, the default timeout is 180 seconds. The managed NIM startup health wait uses a separate 15-minute (900-second) default and still exits early if the container stops before it becomes healthy. On Docker 29.x or hosts using the containerd image store, managed NIM onboarding resolves and pulls the host-platform image digest when NGC exposes a multi-architecture image index. If you still see NGC repository-format or attestation errors, confirm Docker can run docker manifest inspect for the selected image and that you are logged in to nvcr.io. If large prompts still cause timeouts, increase it with NEMOCLAW_LOCAL_INFERENCE_TIMEOUT before re-running onboard:

export NEMOCLAW_LOCAL_INFERENCE_TIMEOUT=300
$$nemoclaw onboard

For local Ollama and vLLM, onboarding retries the container reachability check and can fall back to the host-side health check when the local backend is healthy. If Ollama times out during a cold model load, NemoClaw retries once with a 300-second probe budget before failing. If all attempts fail, the error includes container reachability diagnostics such as HTTP status and host gateway resolution.

NEMOCLAW_LOCAL_INFERENCE_TIMEOUT only covers the inference-server validation probe. The post-create readiness wait has its own budget (NEMOCLAW_SANDBOX_READY_TIMEOUT); refer to Sandbox onboard times out with "did not become ready within Ns" for the readiness path.

Sandbox onboard times out with "did not become ready within Ns"

Onboarding ends with:

  Sandbox 'my-assistant' was created but did not become ready within 180s.
  Retry: $$nemoclaw onboard

This is a separate budget from NEMOCLAW_LOCAL_INFERENCE_TIMEOUT. It covers the readiness wait that follows sandbox creation, including in-sandbox boot, OpenClaw start, and policy load. It does not cover the inference probe. The 180-second default fits typical workstations but can be exceeded when:

• The host is building or uploading the sandbox image for the first time (cold caches, slow link).
• The selected model is large (70B+ parameters or 4-bit/8-bit quantisations that take time to memory-map).
• Onboarding runs on a remote VM where image upload to the gateway streams over the network (for example DGX Station first-run installer).

Raise the budget before re-running onboard:

export NEMOCLAW_SANDBOX_READY_TIMEOUT=600
$$nemoclaw onboard

The variable accepts seconds and applies to the readiness wait only. When the ordinary create deadline expires, NemoClaw tries to delete the partially created sandbox. After successful cleanup, the output ends with Retry: $$nemoclaw onboard. If cleanup fails, NemoClaw instead reports that the failed sandbox could not be removed and prints Manual cleanup: openshell sandbox delete "<name>".

Portable OpenClaw onboarding preserves the sandbox when NemoClaw cannot verify its exact runtime identity. It does not start dashboard forwarding on this failure path. Inspect the preserved sandbox with the status commands below, then follow the recovery guidance from $$nemoclaw <name> status.

The failure path also differs when NemoClaw recreates an OpenShell-managed Docker runtime immediately before this wait. NemoClaw pins the exact OpenShell sandbox ID before recreation. Within the same deadline, NemoClaw requires two consecutive Ready observations that each confirm the exact ID and successful command execution. It retries only OpenShell's exact sandbox is not ready response. If the deadline expires, the ID changes, or another probe fails, NemoClaw preserves diagnostics and attempts to restore the pre-recreation Docker container. If restoration fails, NemoClaw reports that the sandbox and container state is uncertain. NemoClaw does not start dashboard or other host forwarding, and it does not delete a sandbox by its mutable name. It leaves the sandbox in place for inspection and recovery.

If readiness still fails after the extended budget, inspect the gateway and sandbox status:

openshell sandbox list
$$nemoclaw <name> status

If onboarding instead reports that the sandbox "did not re-register with OpenShell after policy application," the same timeout controls that post-policy command-readiness probe. Raise the budget before retrying, then inspect the same gateway and sandbox status if re-registration still fails.

Sandbox onboard fails with "entered Error phase before it became ready"

Onboarding ends with:

  Sandbox 'my-assistant' entered Error phase before it became ready (waited up to 180s).

On a fresh onboard the OpenShell gateway can (re)start its supervisor session and re-register the just-created sandbox. During that window openshell sandbox list briefly reports the sandbox in the transient Error phase before it flips to Ready, as seen on DGX Spark when supervisor restart races the sandbox bootstrap.

NemoClaw polls immediately, starts retrying after 250ms, and backs off to a 2-second cap. It tolerates 30 consecutive Error observations by default so this transient recovers on its own. Only Error that persists through the debounce count is terminal, unless the overall NEMOCLAW_SANDBOX_READY_TIMEOUT deadline expires first. Failed and CrashLoopBackOff are always terminal and fail immediately.

If your host needs more observations for slower re-registration, raise the debounce. Raise NEMOCLAW_SANDBOX_READY_TIMEOUT too if the overall deadline is too short. To fail fast on the first Error poll, set the debounce to 1:

export NEMOCLAW_SANDBOX_READY_ERROR_DEBOUNCE=1
$$nemoclaw onboard

If the failure persists after the debounce, the sandbox is stuck. Inspect the retained diagnostics and gateway state:

openshell sandbox list
$$nemoclaw <name> status

Agent fails at runtime after onboarding succeeds with a compatible endpoint

Some OpenAI-compatible servers (such as SGLang) expose /v1/responses but their streaming mode is incomplete. OpenClaw requires granular streaming events like response.output_text.delta that these backends do not emit.

For the compatible-endpoint provider, NemoClaw now defaults to /v1/chat/completions and skips the Responses API probe entirely unless you opt in. If you onboarded an older release that selected /v1/responses, re-run onboarding so the wizard rebuilds the image with chat completions:

$$nemoclaw onboard

If you previously set NEMOCLAW_PREFERRED_API=openai-responses to force the Responses API, unset it before re-running onboard.

When you enable Telegram messaging with an OpenAI-compatible endpoint, onboarding also checks inference.local from inside the sandbox. If that smoke check fails, fix the compatible-endpoint base URL, credentials, model, or network route before testing the Telegram bot again.

Do not rely on NEMOCLAW_INFERENCE_API_OVERRIDE alone. It patches the config at container startup but does not update the Dockerfile ARG baked into the image. A fresh $$nemoclaw onboard is the reliable fix.

Tool calls appear as assistant text

Local model servers must return structured tool_calls for OpenClaw to dispatch a tool. When the inference response contains only text that resembles a tool request, the gateway treats it as ordinary assistant text and no tool runs. The TUI can display a response such as:

{"arguments":{"query":"robotics"},"name":"memory_search"}

This symptom is different from a network or policy block. $$nemoclaw <name> status, $$nemoclaw <name> logs, and $$nemoclaw debug --quick can all look healthy while conversation-level tool dispatch fails.

Ollama can serve local chat and some simple tool surfaces, but agent loops with several tools, long instructions, or multi-turn dispatch need a server that returns structured tool calls consistently.

Workload	Ollama is usually sufficient	Prefer vLLM with a parser
Plain chat	Yes	Optional
One simple tool with short prompts	Often	Optional
Agent loops with several tools	Risky	Yes
Long system prompts or sender metadata	Risky	Yes
Multi-turn tool dispatch	Risky	Yes

On hosts other than N1x, set up vLLM with automatic tool choice and the tool-call parser that matches the model family for persistent agent use. After the parser-aware server is ready, re-run onboarding. Select the Local vLLM entry marked running (suggested) for a server detected on localhost:8000; for another address, select Other OpenAI-compatible endpoint. On generic hosts, the Local vLLM entry includes an experimental label; on DGX Spark or DGX Station, it does not. On N1x, stop any server that occupies port 8000, then use only the Deferred managed-vLLM preview; existing and compatible servers are not admitted.

Do not rely on direct edits to openclaw.json for a persistent provider change. NemoClaw-managed rebuilds can overwrite those edits, while onboarding keeps the sandbox image, OpenShell route, and host-managed credentials aligned.

Ask the agent to perform an action that requires a tool, then confirm that the TUI does not show a JSON blob as assistant text, the gateway log shows tool dispatch followed by an answer, and $$nemoclaw <name> status reports the intended local vLLM or compatible provider. If JSON still appears as text, confirm that vLLM started with automatic tool choice and the parser required by the model family.

Onboarding fails with duplicate Anthropic message_start events

Validation for an OpenClaw Other Anthropic-compatible endpoint selection ends with an error like:

Anthropic Messages API (streaming): duplicate message_start

For OpenClaw custom Anthropic routes, NemoClaw sends a stream: true request to /v1/messages and validates the SSE event sequence (exactly one message_start, at least one content_block_delta, and a message_stop). This error means the streaming layer on the endpoint or gateway is malformed even though its non-streaming responses are valid. A working non-streaming response does not imply that streaming works. Some inference gateways proxy plain requests correctly but corrupt the SSE stream, for example by emitting message_start twice for one request. OpenClaw uses the streaming path, so without this check the defect would first surface inside the sandbox as a runtime failure.

Hermes and OpenAI-compatible-only agents use the endpoint's /v1/chat/completions surface for custom Anthropic selections instead. Current onboarding validates that surface and does not reject those agents because of a malformed native /v1/messages stream they will not use. An older Hermes sandbox that still uses native Anthropic Messages can report that no final response was produced; re-run onboarding to select and validate the managed Chat Completions route.

Fix the streaming layer on the endpoint or gateway, or onboard with a different Anthropic-compatible endpoint. The official Anthropic provider does not run this check and is not affected. If an OpenClaw sandbox created by an older release fails at runtime with an empty final response on an Anthropic-compatible endpoint, re-run $$nemoclaw onboard so the streaming check can diagnose the endpoint. If the endpoint serves a reasoning-only model, set NEMOCLAW_REASONING=true to skip the streaming check. Streaming defects then surface at runtime instead of during onboarding.

NEMOCLAW_DISABLE_DEVICE_AUTH=1 does not change an existing sandbox

This is expected behavior. NEMOCLAW_DISABLE_DEVICE_AUTH is a build-time setting used when NemoClaw creates the sandbox image. Changing or exporting it later does not rewrite the baked openclaw.json inside an existing sandbox.

If you need a different device-auth setting, rerun onboarding so NemoClaw rebuilds the sandbox image with the desired configuration. For the security trade-offs, refer to Security Best Practices.

openclaw.json is empty after changing inference

Some runtime inference changes can leave /sandbox/.openclaw/openclaw.json empty if the write fails partway through. When that happens, OpenClaw commands may report that the config is empty instead of showing a raw JSON parse error.

Current NemoClaw sandboxes capture a known-good config baseline after a successful startup. On the next sandbox startup, NemoClaw restores openclaw.json from OpenClaw's last-good copy when available, or from the NemoClaw baseline. Recovery validates the config tree and selected source before atomically replacing openclaw.json and .config-hash. If it detects an unsafe link, unexpected owner, or path change, startup fails closed without following or modifying the unsafe target. If the sandbox still cannot start or reports that no baseline is available, rebuild it from the host:

$$nemoclaw <name> rebuild

A Shields command reports corrupt persisted state

If shields up, shields down, or shields status reports corrupt persisted state, NemoClaw refuses to infer or change the Shields posture. It preserves the Shields state file, timer marker, and transition record so an active recovery authority remains intact.

While the state remains corrupt, do not use shields up or an ordinary rebuild to replace it. Inspect the reported state-file error and restore the state file from a trusted host backup before retrying.

shields up or shields down fails after .config-hash was removed

/sandbox/.openclaw/.config-hash is the integrity sidecar for openclaw.json; deleting it during a manual config edit removes the file the shields guard captures alongside the config. From the default mutable posture, $$nemoclaw <name> shields up regenerates a stale hash from the current openclaw.json bytes. On sandboxes with the updated guard, the same command synthesizes a truly absent .config-hash under the frozen tree. Only a truly absent file is repaired; an unexpected file type at that name still fails closed. $$nemoclaw <name> shields down does not synthesize the hash: with the file missing it fails closed without modifying the config. If shields are already up, another shields up also fails closed when the hash is missing. Do not use mutable-posture synthesis to recover a locked sandbox.

Sandboxes created by an older NemoClaw release keep the older guard baked into the container image, where a missing .config-hash makes the transition fail closed and quarantine openclaw.json by renaming it to .nemoclaw-rejected-openclaw.json-<random> in the same directory. The config bytes are preserved, not deleted. Upgrade the NemoClaw CLI before either recovery path because an older CLI restages the older guard. To preserve the quarantined settings, copy the file to the host before rebuilding:

docker exec <container> ls -a /sandbox/.openclaw
docker cp <container>:/sandbox/.openclaw/<quarantine-file> ./openclaw.json.recovered
$$nemoclaw <name> rebuild --yes

After the rebuild, inspect ./openclaw.json.recovered and reapply required settings with the host-side config set command. Do not overwrite the regenerated openclaw.json with an unreviewed quarantine copy. To discard the quarantined settings, upgrade the CLI and run $$nemoclaw <name> rebuild --yes without copying the file.

openclaw channels add or remove is blocked inside the sandbox

This is expected. The messaging channel list is frozen into the sandbox's container image when the image is built during $$nemoclaw onboard or $$nemoclaw rebuild (the selected channel names are passed to the docker build as NEMOCLAW_MESSAGING_CHANNELS_B64 and written into agent config such as /sandbox/.openclaw/openclaw.json for OpenClaw or /sandbox/.hermes/.env for Hermes). Changes made inside the running sandbox do not persist across rebuilds, so openclaw channels commands that mutate the config are intercepted. NemoClaw's sandbox entrypoint installs a guard that intercepts openclaw channels <add|remove> and prints an actionable error pointing at the host-side commands below, instead of letting the call fail deep in the binary with a raw EACCES trace.

Run the equivalent host-side command instead:

$$nemoclaw <sandbox> channels list
$$nemoclaw <sandbox> channels add <telegram|discord|slack|wechat|whatsapp>
$$nemoclaw <sandbox> channels remove <telegram|discord|slack|wechat|whatsapp>

channels add registers credentials with the OpenShell gateway and channels remove clears them. Both offer to rebuild the sandbox so the image reflects the new channel set. In non-interactive mode (NEMOCLAW_NON_INTERACTIVE=1, or any run without a terminal on stdin), the commands stage the change and leave the rebuild to a follow-up $$nemoclaw <sandbox> rebuild. WeChat and WhatsApp are experimental. Review Choose Messaging Channels before enabling them.

WeChat captures its bot token through a host-side QR scan during $$nemoclaw onboard or channels add wechat. You scan the iLink QR from WeChat on your phone and NemoClaw registers the captured token with the OpenShell gateway.

WhatsApp pairs entirely inside the sandbox. NemoClaw advertises WhatsApp for OpenClaw and Hermes sandboxes after you add the channel on the host. Run openclaw channels login --channel whatsapp inside OpenClaw sandboxes, or run hermes whatsapp inside Hermes sandboxes.

scripts/rcf_patch.py is missing from the blueprint

scripts/rcf_patch.py is intentionally absent from current NemoClaw blueprints. Older QA plans used that helper for a Dockerfile "Patch-4" test that corrupted the build-time replaceConfigFile monkey-patch and expected ERROR: Patch 4 (replaceConfigFile EACCES) not applied. The old Patch-4 fail-closed test no longer applies because NemoClaw no longer patches OpenClaw's compiled replaceConfigFile source at image build time.

Current sandboxes use a mutable-default config model instead. Before a reviewed host-side lockdown, /sandbox/.openclaw/openclaw.json is group-writable by the sandbox and gateway users, so OpenClaw config mutations should write normally rather than requiring an EACCES swallow. After lockdown, runtime config mutations should fail cleanly or route users to the supported host-side NemoClaw command.

To validate this area now, use the config lifecycle tests instead of looking for rcf_patch.py:

npm run build:cli
npm test -- test/repro-2681-group-writable.test.ts

openclaw config set or unset is blocked inside the sandbox

This is expected. NemoClaw builds the sandbox's OpenClaw configuration (/sandbox/.openclaw/openclaw.json) from host-side onboarding, rebuild, inference, policy, and messaging inputs. Fresh sandboxes keep that file writable by default so the agent can manage runtime state, but direct in-sandbox edits are not the supported or durable path for NemoClaw-managed settings. NemoClaw's sandbox entrypoint installs a guard that intercepts openclaw config set and openclaw config unset and prints an actionable error, because changes made inside the running sandbox do not persist across rebuilds.

For most configuration changes, exit the sandbox and rerun onboarding:

$$nemoclaw onboard

If NemoClaw reports a resumable failed onboarding session, run $$nemoclaw onboard --resume instead. This rebuilds the sandbox with your updated settings.

For advanced live edits, use the host-side config command instead of running openclaw config set inside the sandbox:

$$nemoclaw <sandbox> config set --key <dotpath> --value '<json-or-string>' --restart

Host-side config set validates any HTTP or HTTPS URLs in the new value, including URLs nested inside JSON objects or arrays. NemoClaw rejects loopback, private, reserved, and internal hosts; DNS names must resolve successfully and must not resolve to private/internal addresses. HTTP URLs are written with the validated IP address pinned to reduce DNS-rebinding risk. Avoid putting credentials in config values; rotate provider credentials with the credential-management commands instead.

openclaw doctor --fix cannot repair Discord channel config inside the sandbox

This is expected in NemoClaw-managed sandboxes. NemoClaw bakes channel entries into /sandbox/.openclaw/openclaw.json at image build time.

As a result, commands that try to rewrite the baked config from inside the sandbox, including openclaw doctor --fix, cannot repair Discord, Telegram, or Slack channel entries in place.

If your Discord channel config is wrong, rerun onboarding so NemoClaw rebuilds the sandbox image with the correct messaging selection. Do not treat a failed doctor --fix run as proof that the Discord gateway path itself is broken.

If openclaw doctor reports that it moved Telegram single-account values under channels.telegram.accounts.default, rerun onboarding and rebuild the sandbox rather than trying to patch openclaw.json in place. Current NemoClaw rebuilds bake Telegram in the account-based layout and set Telegram group chats to groupPolicy: open, which avoids the empty groupAllowFrom warning path for default group-chat access.

openclaw doctor --fix tightened config permissions and the gateway can no longer save config

In a mutable NemoClaw sandbox, the gateway UID and the sandbox UID share the sandbox group, so /sandbox/.openclaw is setgid and group-writable (2770) and openclaw.json is group-writable (660). OpenClaw's openclaw doctor --fix enforces its own single-user 700/600 layout, so running it inside the sandbox strips group write and breaks gateway-side config writes (for example, control-UI toggles that mutate openclaw.json).

When you invoke the command through the host-side one-shot path, NemoClaw restores the mutable contract as the command exits:

$$nemoclaw <sandbox> exec -- openclaw doctor --fix

When cleanup succeeds, exec preserves the openclaw doctor --fix exit code. If cleanup cannot inspect, restore, or verify the mutable config permission contract, it returns a cleanup failure instead and prints OpenClaw permission cleanup failed (...) with the command and cleanup statuses. Do not recursively change ownership or permissions over a config tree that failed this safety check. Inspect the reported condition and rebuild from trusted host-side configuration if the tree or image boundary is not the expected NemoClaw layout.

If you ran openclaw doctor --fix from an interactive connect shell or use an older sandbox image, repair the mutable contract without rebuilding:

$$nemoclaw <sandbox> doctor --fix

$$nemoclaw <sandbox> doctor reports the drift as a Config permissions warning, and --fix restores 2770/660. Restarting the sandbox repairs the same drift automatically when the config tree passes its safety checks, and NemoClaw's own rebuild re-applies the contract after its post-upgrade openclaw doctor --fix step. For a persisted root-owned 700/600 tree, startup reclaims ownership only when both fixed config files have that exact posture under the expected sandbox-owned parent. Other root-owned layouts, links, mounts, and ambiguous metadata fail closed so startup cannot mistake a shields-locked or unsafe tree for mutable drift. If startup reports [SECURITY] Refusing mutable config permission normalization, NemoClaw stops startup without following or modifying the unsafe target; safe permission repairs completed before detection are not rolled back. Rebuild with the current image and trusted host-side configuration instead of repairing the tree recursively.

When verifying gateway write access by hand, step down to the gateway UID with the image's installed mechanism so the sandbox group membership is initialized:

setpriv --reuid=gateway --regid=gateway --init-groups -- sh -c 'echo ok >> /sandbox/.openclaw/openclaw.json'

If setpriv is unavailable, rebuild the sandbox from a NemoClaw-managed image that includes util-linux. When a root entrypoint must change identity, it fails closed if this required privilege-separation command is missing.

Do not probe with su -s /bin/sh gateway ...: su does not initialize the gateway's supplementary groups the same way, so a group-write probe can spuriously report EACCES even when the mutable contract is intact.

A NemoClaw sandbox has two intentional permission states for /sandbox/.openclaw; 700/600 is not one of them:

• Mutable default: /sandbox/.openclaw is 2770 sandbox:sandbox and openclaw.json is 660 sandbox:sandbox. Both the sandbox user and the gateway (same sandbox group, different UID) can write config, so control-UI toggles persist.
• Host-locked state: openclaw.json is read-only for in-sandbox writers and the config dir is owned by root, with the immutable bit set where available. No in-sandbox writes are expected; use the host-side $$nemoclaw <sandbox> config set flow described in openclaw config set or unset is blocked inside the sandbox.
• 700/600 (drift): the layout that upstream openclaw doctor --fix imposes inside a mutable sandbox. It is not a supported NemoClaw state; recover with $$nemoclaw <sandbox> doctor --fix or a sandbox restart.

Discord bot logs in, but the channel still does not work

Separate the problem into two parts:

1. Baked config and provider wiring

   Check that onboarding selected Discord and that the sandbox was created with the Discord messaging provider attached. If Discord was skipped during onboarding, rerun onboarding and select Discord again.

2. Native Discord gateway path

   Successful login alone does not prove that Discord works end to end. Discord also needs a working gateway connection to gateway.discord.gg. If logs show errors such as getaddrinfo EAI_AGAIN gateway.discord.gg, repeated reconnect loops, or a 400 response while probing the gateway path, the problem is usually in the native gateway/proxy path rather than in the baked config.

Common signs of a native gateway-path failure:

• REST calls to discord.com succeed, but the Discord channel never becomes healthy
• gateway.discord.gg fails with DNS resolution errors
• the WebSocket path returns 400 instead of opening a tunnel
• native command deployment fails even though the bot token itself is valid

In that case:

• keep the Discord policy preset applied
• verify the sandbox was created with the Discord provider attached
• inspect gateway logs and blocked requests with openshell term
• treat the failure as a native Discord gateway problem, not as a bridge startup problem

Discord preset validation behind a proxy

The built-in Discord policy preset intentionally allows the Node binaries used by the messaging runtime and does not allow curl. As a result, curl -s https://discord.com failing, hanging, or printing no output is not proof that the Discord preset is broken.

Behind the OpenShell proxy, direct DNS-only checks can also be the wrong signal. For example, dns.resolve("gateway.discord.gg") can fail even when HTTPS requests routed through the proxy are healthy.

Use Node HTTPS as the manual REST probe:

node - <<'NODE'
const https = require("node:https");

https
  .get("https://discord.com/api/v10/gateway", (res) => {
    console.log(`${res.statusCode} ${res.statusMessage || ""}`.trim());
    res.resume();
  })
  .on("error", (err) => {
    console.error(err.message);
    process.exitCode = 1;
  });
NODE

To check Discord CDN egress, use the same Node HTTPS path:

node - <<'NODE'
const https = require("node:https");

https
  .get("https://cdn.discordapp.com/", (res) => {
    console.log(`${res.statusCode} ${res.statusMessage || ""}`.trim());
    res.resume();
  })
  .on("error", (err) => {
    console.error(err.message);
    process.exitCode = 1;
  });
NODE

Any HTTP status from these probes means the Node process reached the endpoint; the exact status can vary by unauthenticated path. If the Node REST probe works but the Discord channel is still unhealthy, investigate the native gateway path instead of widening the preset. Check the gateway logs and blocked-request output with openshell term, and look for gateway.discord.gg connection or WebSocket upgrade failures.

Messaging bridge appears running but no messages arrive

Telegram getUpdates allows only one active poller per bot token. Reusing Discord or Slack credentials can create competing gateway or Socket Mode sessions and unreliable message delivery. $$nemoclaw status can still report a bridge as running because the gateway process itself is alive.

For Telegram group chats, first check BotFather privacy mode. New Telegram bots default to privacy mode enabled, which prevents group messages from reaching getUpdates even when the user mentions the bot. In @BotFather, run /setprivacy, choose the bot, and choose Disable. Then remove the bot from the affected group and add it back; Telegram applies the privacy-mode change to group delivery only after the bot rejoins.

For Telegram direct messages, make sure the rebuilt sandbox has a DM allowlist. Set TELEGRAM_ALLOWED_IDS before rebuild; TELEGRAM_AUTHORIZED_CHAT_IDS and TELEGRAM_CHAT_ID are accepted as compatibility aliases. Keep the aliases until QA automation and public repro templates have stopped exporting them for at least one full release. Bot API sendMessage sends from the bot to a chat, so it only proves outbound Telegram API access. To prove inbound agent routing, send a message from the Telegram client as an allowed user and then watch the gateway log for the agent turn and outbound reply. For a reproducible outbound runtime check, run NEMOCLAW_RUN_LIVE_E2E=1 npx vitest run --project e2e-live test/e2e/live/messaging-providers.test.ts --silent=false --reporter=default with NVIDIA_INFERENCE_API_KEY set. The check imports the installed OpenClaw Telegram runtime-api.js, calls sendMessageTelegram through an OpenShell-rewritten credential against a host-side fake Telegram API, and verifies the captured chat, text, token rewrite, and absence of unresolved placeholders. When TELEGRAM_BOT_TOKEN_REAL and TELEGRAM_CHAT_ID_E2E are also set, the same lane performs an additional real outbound send; it does not prompt for or claim an interactive inbound reply.

To diagnose, open a shell in the sandbox and inspect the gateway log:

$$nemoclaw <sandbox-name> connect
tail -f /tmp/gateway.log

A repeating line like the following confirms the conflict:

[telegram] getUpdates conflict: 409: Conflict: terminated by other getUpdates request; retrying in 30s.

To fix, run $$nemoclaw <other-sandbox> destroy on whichever sandbox should stop polling, or rerun onboarding on it with the channel disabled. NemoClaw checks only the sandboxes in the selected OpenShell gateway's sandbox registry. It cannot detect or prevent Slack credential reuse across independent OpenShell gateways. Run only one active Slack sandbox on each OpenShell gateway. Use distinct Slack bot and app tokens for Slack sandboxes on different OpenShell gateways. Within the selected registry, onboarding, rebuild, and channels add abort on a conflict or an incomplete required check, including unavailable credential hashes. Only channels add <channel> --force can accept the duplicate-consumer or shared-resource risk. Sandboxes created before these checks were added, or managed by independent gateways, may still have a conflict without a NemoClaw warning.

Landlock filesystem restrictions silently degraded

After sandbox creation, NemoClaw checks whether the host kernel supports Landlock (Linux 5.13+). If the kernel is too old or you are running on macOS (where the Docker VM kernel may lack Landlock), a warning prints:

⚠ Landlock: Docker VM kernel <version> does not support Landlock (requires ≥5.13).
  Sandbox filesystem restrictions will silently degrade (best_effort mode).

This warning is informational and does not block sandbox creation. The sandbox runs without kernel-level filesystem restrictions, relying on container mount configuration instead. For full filesystem enforcement, run on a Linux kernel 5.13 or later (Ubuntu 22.04 LTS and later include Landlock support).

Landlock filesystem policy blocks sandbox startup

Deep Agents uses strict Landlock compatibility. If the host kernel, Docker VM, or sandbox filesystem mount cannot enforce the managed read-only policy, OpenShell refuses to start the sandbox instead of silently degrading.

Run Deep Agents on a Linux kernel 5.13 or later with a container runtime that exposes Landlock to the sandbox. After moving to a compatible host or runtime, rerun onboarding or rebuild the sandbox:

nemo-deepagents <name> rebuild

Sandbox lost after gateway restart

Sandboxes created with OpenShell versions older than 0.0.24 can become unreachable after a gateway restart because SSH secrets were not persisted. Running $$nemoclaw onboard automatically upgrades OpenShell to 0.0.24 or later during the preflight check. After the upgrade, recreate the sandbox with $$nemoclaw onboard.

DNS-backed HTTPS endpoint is not supported

NemoClaw rejects an explicit custom endpoint when it resolves a public HTTPS hostname but cannot pin the same peer address across the downstream OpenShell runtime boundary while preserving TLS SNI and host validation. This can appear during a direct blueprint run, custom-endpoint onboarding, or a host-side config set write. It does not appear during a runtime $$nemoclaw inference set switch on an already-onboarded sandbox; that command routes a DNS-backed HTTPS endpoint through a local HTTPS Pin Runtime adapter instead of rejecting it. Refer to Commands for details.

Use an HTTPS IP-literal endpoint whose certificate is valid for that address. If your deployment permits non-TLS provider traffic, you can instead use a public HTTP endpoint that NemoClaw can rewrite to a DNS-pinned address. Do not bypass the check with a private or internal address or by editing the persisted sandbox config directly. For the full endpoint rules, refer to Meet Custom Endpoint Security Requirements.

Agent cannot reach external hosts through a proxy

NemoClaw uses a default proxy address of 10.200.0.1:3128 (the OpenShell-injected gateway). If your environment uses a different proxy, set NEMOCLAW_PROXY_HOST and NEMOCLAW_PROXY_PORT before onboarding:

export NEMOCLAW_PROXY_HOST=proxy.example.com
export NEMOCLAW_PROXY_PORT=8080
$$nemoclaw onboard

These are build-time settings baked into the sandbox image. Changing them after onboarding requires re-running $$nemoclaw onboard to rebuild the image.

When HTTP_PROXY or HTTPS_PROXY is set on the host, NemoClaw adds localhost, 127.0.0.1, ::1, 0.0.0.0, the container-host aliases host.docker.internal and host.containers.internal, and the managed inference hostname inference.local to NO_PROXY for host-side subprocesses and for the env forwarded into openshell sandbox create. This keeps local Ollama health checks, model pulls, and managed inference traffic from being chained through a corporate or desktop proxy at the sandbox-create boundary, while preserving the proxy for external hosts. For the local provider validation probe, NemoClaw removes HTTP_PROXY, HTTPS_PROXY, and ALL_PROXY from the probe process and sets NO_PROXY=* instead. A host proxy therefore cannot answer for the local endpoint, including the host.docker.internal alias used for Windows-host Ollama. Inside the running sandbox, processes continue to use the OpenShell L7 proxy for inference.local so OpenShell's internal routing, DNS, and audit boundaries stay intact.

Agent cannot reach a host-side HTTP service

When a sandbox needs to call an HTTP service running on the host, use the normal OpenShell network policy path. Expose the service on a host IP address that the OpenShell gateway can reach, create a custom NemoClaw policy preset for that IP and port, and apply it with $$nemoclaw <sandbox> policy add --from-file. The sandbox request then flows through the OpenShell proxy while NemoClaw preserves the existing live policy entries.

Do not rely on host.docker.internal or host.openshell.internal as a general-purpose host-service path. Those names may appear in the sandbox's /etc/hosts, but in OpenShell's sandbox network they are not guaranteed to point at a reachable host gateway. Bypassing the proxy with --noproxy '*' also bypasses network policy enforcement and audit.

First, make sure the host-side service listens on a non-loopback address. For example, a health endpoint on port 50001 should be reachable from the host IP, not only from 127.0.0.1:

curl -s http://10.0.0.5:50001/health

Expected output:

{"status":"ok"}

Then create a custom NemoClaw preset for the host-side service. Replace 10.0.0.5, 50001, paths, methods, and binaries with the service you want the sandbox to reach:

preset:
  name: host-memory-api
  description: "Host memory API"
network_policies:
  host_memory_api:
    name: host_memory_api
    endpoints:
      - host: 10.0.0.5
        port: 50001
        protocol: rest
        enforcement: enforce
        rules:
          - allow: { method: GET, path: "/health" }
    binaries:
      - { path: /usr/bin/curl }

Apply the preset to the running sandbox with the NemoClaw CLI:

$$nemoclaw my-assistant policy add --from-file ./host-memory-api.yaml

After you apply the policy, retry the request from inside the sandbox without disabling the proxy:

curl -s http://10.0.0.5:50001/health

Expected output:

{"status":"ok"}

If the request is still denied, check the blocked request in openshell term. The policy binaries list must include the executable path that actually made the request. If the response changes from policy_denied to upstream_unreachable, the policy matched, but the OpenShell gateway could not reach the host IP and port.

Agent cannot reach an external host

OpenShell blocks outbound connections to hosts not listed in the network policy. Open the TUI to see blocked requests and approve them:

openshell term

To permanently allow an endpoint, add it to the network policy. Refer to Customize the Network Policy for details. For Deep Agents, follow Customize the Network Policy to choose between built-in presets, reviewed custom presets, baseline edits, and live-policy replacement.

Dashboard not reachable after setting a custom port

If you ran $$nemoclaw onboard with a custom dashboard port and onboarding completed but the dashboard URL is unreachable, the sandbox was most likely created with an older NemoClaw version that did not pass the dashboard port into the sandbox at startup. The browser may show connection refused or fail to load the page. The gateway inside the sandbox continued listening on the default port 18789 while the SSH tunnel forwarded the custom port, leaving nothing at the other end of the tunnel.

Re-run onboarding on the current NemoClaw release with the desired port. Current versions derive the dashboard port from CHAT_UI_URL automatically and inject it into the sandbox:

CHAT_UI_URL=http://127.0.0.1:19000 $$nemoclaw onboard

If you need to run multiple sandboxes at different ports at the same time, refer to Running multiple sandboxes simultaneously.

Control UI config endpoint returns 404 or non-JSON

The Control UI loads its runtime configuration from a gateway endpoint, not from a static controlui.bootstrap.config.json file. No controlui.bootstrap.config.json path is served, so requesting it returns HTTP 404 Not Found with a short plain-text body, and piping that response to jq fails with a parse error such as Invalid numeric literal.

The supported Control UI config endpoint is /__openclaw/control-ui-config.json, served by the OpenClaw gateway on the forwarded dashboard port. It is gated by the gateway auth token:

• An unauthenticated request returns HTTP 401 Unauthorized with a JSON body ({"error":{"message":"Unauthorized","type":"unauthorized"}}), which is already valid JSON.
• An authenticated request returns HTTP 200 OK with the Control UI config as JSON.

Resolve the forwarded dashboard port, then authenticate with the gateway token from $$nemoclaw <name> gateway-token:

openshell forward list                       # note the dashboard PORT for the sandbox
export DASH_PORT=<port>
TOKEN=$($$nemoclaw <name> gateway-token --quiet)
curl -fsS -H "Authorization: Bearer $TOKEN" \
  "http://127.0.0.1:${DASH_PORT}/__openclaw/control-ui-config.json" | jq empty \
  && echo "Control UI config is valid JSON"

The token is sensitive; treat it like a password and do not log, share, or commit it. For browser access, use the tokenized URL from $$nemoclaw <name> dashboard-url instead of calling the config endpoint directly.

Hermes manages its own dashboard sessions and does not expose an OpenClaw gateway auth token or a /__openclaw/control-ui-config.json endpoint. Use nemohermes <name> status to see the dashboard and API endpoints for a Hermes sandbox.

Ollama auth proxy did not start

NemoClaw keeps Ollama bound to 127.0.0.1:11434 and starts a token-gated reverse proxy on 0.0.0.0:11435 so the sandbox can reach Ollama without exposing it to the local network. If the proxy fails to start, onboarding exits before configuring inference.

Check whether the proxy port is occupied by another process:

sudo lsof -i :11435

Stop the conflicting process and re-run $$nemoclaw onboard. The wizard cleans up stale proxy processes from previous runs automatically, so most failures resolve by retrying.

The proxy token is persisted to ~/.nemoclaw/ollama-proxy-token with 0600 permissions. If the file is missing or unreadable after a host reboot, re-running $$nemoclaw onboard regenerates it.

Ollama auth proxy is unreachable from the sandbox

On native Linux Docker-driver hosts, a host firewall can allow the host proxy check but block sandbox traffic to the Ollama auth proxy. When that happens, onboarding exits before it saves the inference route and prints output like:

✗ Sandbox containers cannot reach the Ollama auth proxy at host.openshell.internal:11435.
  A host firewall may be blocking traffic from the OpenShell Docker bridge.

Apply the ufw command printed by onboarding, then rerun onboarding. If the message does not include a subnet, derive it from the OpenShell Docker network:

SUBNET=$(docker network inspect openshell-docker --format '{{(index .IPAM.Config 0).Subnet}}')
sudo ufw allow from "$SUBNET" to any port 11435 proto tcp
$$nemoclaw onboard

Docker Desktop, WSL, and hosts without the OpenShell Docker network use different routing models. In those cases NemoClaw treats an unavailable sandbox-side probe as non-blocking and relies on the regular proxy health check.

host.docker.internal does not reliably reach the host from the sandbox

Configuring an inference provider with a base URL like http://host.docker.internal:11434/v1 does not reliably reach a host Ollama service from inside the OpenShell sandbox. OpenShell runs sandboxes inside a k3s network, where host.docker.internal is not a portable host-service route. Depending on the platform, it may fail DNS resolution or resolve to an internal gateway/bridge address where the host's port 11434 is not forwarded. The sandbox then sees a DNS failure or connection refused:

getent hosts host.docker.internal

Expected output:

172.17.0.1      host.docker.internal host.openshell.internal

no_proxy=host.docker.internal curl -v http://host.docker.internal:11434/api/tags

Expected output:

* connect to 172.17.0.1 port 11434 failed: Connection refused

For local Ollama, use the auth-proxy URL that NemoClaw's "Local Ollama" onboard option configures automatically:

http://host.openshell.internal:11435/v1

host.openshell.internal resolves to the same gateway IP, and the token-gated Ollama auth proxy binds port 11435 there and forwards requests to 127.0.0.1:11434 on the host. If you need a different host service exposed to the sandbox, route it through the OpenShell gateway rather than relying on host.docker.internal. Refer to issue #3136.

Local inference health check resolves to IPv6

Local inference health checks now use 127.0.0.1 instead of localhost. On systems where localhost resolves to ::1 first, older NemoClaw releases could probe the wrong address and report the local backend as unreachable even when it was running. If you see this on a current NemoClaw release, verify that the local backend binds an IPv4 address and not only ::1.

Blueprint run failed

View the error output for the failed blueprint run:

$$nemoclaw <name> logs

Use --follow to stream logs in real time while debugging.

DGX Spark

For an end-to-end walkthrough with local inference on DGX Spark, refer to the NVIDIA Spark playbook.

Host freezes or logs NVRM NV_ERR_NO_MEMORY under local vLLM load

Treat a full host freeze separately from an agent tool-call hang. If the Spark stops responding to SSH and ping, and the journal contains NVRM NV_ERR_NO_MEMORY or no software-side crash record, first isolate the local inference server before changing MCP or network policy configuration. For onboarding-time context, refer to Use an Existing Server.

Check whether vLLM is a bring-your-own server or the NemoClaw managed Spark profile:

docker ps --format 'table {{.Names}}\t{{.Image}}\t{{.Status}}\t{{.Ports}}'
curl -sf http://127.0.0.1:8000/v1/models
free -h
journalctl -k --since "24 hours ago" --no-pager | grep -Ei 'NVRM|OOM|out of memory|lockup|watchdog'

For an existing vLLM server, inspect its launch arguments:

docker inspect <vllm-container> --format '{{json .Config.Cmd}}'

Large checkpoints without explicit quantization, very long --max-model-len values, high --gpu-memory-utilization, and multiple concurrent sequences all consume the Spark's shared CPU/GPU memory pool. Before reintroducing agent tools, restart vLLM with a smaller envelope, for example:

vllm serve <model> \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.75 \
  --max-num-seqs 1 \
  --max-num-batched-tokens 4096

If the host still logs NVRM NV_ERR_NO_MEMORY while loading the model, switch to a smaller or quantized checkpoint. For managed setup, prefer NEMOCLAW_PROVIDER=install-vllm, which selects the Spark profile and its registered model-specific serve arguments. After standalone vLLM is stable, re-run onboarding and add MCP servers back one group at a time.

CoreDNS CrashLoop after onboarding

If CoreDNS in the embedded k3s cluster crashes shortly after setup, it is usually because it resolves against 127.0.0.11, which does not route inside the gateway container. Run fix-coredns.sh to point CoreDNS at the container gateway IP instead, then recreate the sandbox.

k3s cannot find a freshly built image

After building a new sandbox image, k3s inside the gateway container sometimes fails to pull it even though the image exists on the host. Remove the gateway registration, stop any leftover host gateway process, then re-run setup.

openshell gateway remove nemoclaw
sudo pkill -f openshell-gateway
$$nemoclaw onboard --resume

GPU passthrough on Spark

GPU passthrough is not CI-tested on DGX Spark. It is expected to work when you pass --gpu and the NVIDIA Container Toolkit is configured. Verify the toolkit is configured by running docker run --rm --runtime=nvidia --gpus all nvidia/cuda:12.8.0-base-ubuntu24.04 nvidia-smi from the host. If nvidia-smi works on the host but onboarding says GPU passthrough was not enabled, install or repair the NVIDIA Container Toolkit, then run sudo nvidia-ctk runtime configure --runtime=docker && sudo systemctl restart docker. If a reusable gateway was previously started without GPU passthrough, NemoClaw replaces it automatically only when no other registered sandboxes depend on it, or when --recreate-sandbox is recreating the only registered sandbox with the same name. When shared gateway cleanup would be unsafe, follow the targeted destroy or gateway-removal commands printed by onboarding.

unresolvable CDI devices nvidia.com/gpu=all during gateway start

Recent NVIDIA Container Toolkit installs configure the Docker daemon for Container Device Interface (CDI) device injection, which a GPU-enabled gateway start then auto-selects. If no nvidia.com/gpu CDI spec has been generated on the host yet, gateway start fails with Docker responded with status code 500: CDI device injection failed: unresolvable CDI devices nvidia.com/gpu=all. Outside Station Express, the standard NemoClaw installer detects this gap before onboarding, first tries to enable the NVIDIA CDI refresh systemd units, and can fall back to generating the spec directly with nvidia-ctk. Station Express never falls back to direct CDI generation. The generic Ubuntu, Colossus BaseOS, and exact AI Developer Tools paths require the packaged refresh lifecycle to work; if it fails or omits nvidia.com/gpu=all, inspect nvidia-cdi-refresh.service, repair it, and rerun the printed exact-commit install command. Other factory-runtime profiles stop when the CDI device is missing without enabling or restarting the refresh units. If you run $$nemoclaw onboard directly, preflight prints the manual remediation instead. The native Linux fix is the same on Docker hosts whose docker info advertises a non-empty CDISpecDirs. On WSL with Docker Desktop, Docker may advertise CDI directories even though --device nvidia.com/gpu=all is not usable from the WSL distro. For that runtime, NemoClaw skips Linux CDI repair and uses Docker's --gpus compatibility path for sandbox GPU access. This compatibility path can be retired once Docker Desktop exposes usable nvidia.com/gpu CDI specs inside WSL, or once OpenShell no longer requires host-visible CDI specs for Docker Desktop WSL GPU passthrough.

Enable the refresh units, verify they list nvidia.com/gpu entries, then rerun onboarding:

sudo systemctl enable --now nvidia-cdi-refresh.path nvidia-cdi-refresh.service
nvidia-ctk cdi list
$$nemoclaw onboard

For other native Linux installations, if the refresh units are unavailable or do not generate CDI devices, generate the spec directly:

sudo mkdir -p /etc/cdi
sudo nvidia-ctk cdi generate --output=/etc/cdi/nvidia.yaml
nvidia-ctk cdi list

On WSL with Docker Desktop, confirm Docker Desktop WSL integration is enabled for your distro and verify Docker GPU access from WSL:

docker run --rm --gpus all nvcr.io/nvidia/k8s/cuda-sample@sha256:7c7540bdf1f942d4fb6db97069fd6c289471b54ac29e3c7fcdf914cf77af7d41

If GPU passthrough is not required on this host, rerun onboarding with --no-gpu instead.

GPU routing or compatibility patch failed

The route depends on the host environment and the operator control. Identify the matching path before applying the recovery guidance.

Do not apply this compatibility guidance to portable OpenClaw onboarding. That path requires native OpenShell GPU injection and does not use NEMOCLAW_DOCKER_GPU_PATCH, so do not set fallback, 1, or another legacy nonzero value for a portable OpenClaw run.

Symptom	Route or stage	Recovery
Native --gpu is rejected, host runtime evidence identifies GPU injection failure, or an explicit driver proof fails and host configuration confirms no GPU attachment	Ordinary Linux native attempt	The default native-only route stops. Retry with NEMOCLAW_DOCKER_GPU_PATCH=fallback only if you explicitly accept one bounded compatibility retry, or use =1 to select compatibility before creation.
Cleanup could not be proven safe	Native-to-compatibility handoff	Run the printed sandbox deletion command, verify both the gateway row and OpenShell-managed Docker containers labeled for that sandbox are absent, then rerun onboarding.
The patched container exits or the compatibility attempt fails	Compatibility recreation	Inspect the saved diagnostics and the rollback outcome, then repair the NVIDIA Container Toolkit/CDI configuration. Keep the sandbox when the pre-patch container was restored. Use only an exact-container cleanup command printed after rollback. If no command was printed, inspect the sandbox and its labeled containers before removing anything. Then rerun onboarding.
A recreated container inherits only a loopback DNS stub and no usable upstream	Compatibility DNS fallback	Repair the host's systemd-resolved upstream configuration, then rerun onboarding.

For bridge-networked compatibility recreation without an explicit container DNS setting, NemoClaw selects a usable IPv4 upstream from systemd-resolved and probes that exact --dns path before it stops the original container. If the probe confirms that the resolver is unreachable, recreation stops and leaves the original container in place. An IPv6-only upstream list does not become a compatibility override; NemoClaw preserves Docker's default resolver path instead. Containers with explicit DNS settings or host networking keep their existing DNS path and do not use the fallback probe.

Ordinary native Linux bounded fallback

Ordinary Linux GPU onboarding uses native OpenShell GPU injection and stops on failure by default. Unset, auto, and 0 all preserve this native-only confinement boundary. NEMOCLAW_DOCKER_GPU_PATCH=fallback is the explicit operator authorization for one bounded retry. With that control set, if sandbox creation rejects the native GPU flag before progress, the exact OpenShell-managed container labeled for that sandbox records a host runtime GPU-injection error, or an explicit nvidia-smi driver proof fails while that container's immutable host configuration confirms that no GPU was attached, NemoClaw captures redacted diagnostics, deletes the incomplete sandbox, verifies that no OpenShell-managed Docker container labeled for that sandbox remains, and retries exactly once through the compatibility path. Free-form build/list text and sandbox-reported CUDA output never independently authorize the broader retry. Without corroborating host evidence, onboarding fails closed even when fallback is set and directs the operator to clean up and explicitly select compatibility with NEMOCLAW_DOCKER_GPU_PATCH=1 if desired. Before the authorized retry, NemoClaw warns that the legacy GPU compatibility envelope recreates the OpenShell-managed Docker container and may relax container confinement compared with native injection. Specifically, compatibility recreation adds SYS_PTRACE, adds apparmor=unconfined when the original container has no AppArmor option, and uses a compatibility policy that makes /proc writable for the NVIDIA runtime's process-name initialization. These broader settings are why onboarding warns before the swap and retains a native-only opt-out. NemoClaw verifies cleanup with two stable checks (that sandbox is absent from the gateway list and no OpenShell-managed Docker containers labeled for that sandbox remain) before retrying through the compatibility path. Cleanup is polled at most five times, one second apart, and both conditions must pass twice consecutively; otherwise onboarding stops before the retry. These fail-closed safety limits are the internal constants STABLE_ABSENCE_CHECKS (2), MAX_CLEANUP_ATTEMPTS (5), and CLEANUP_POLL_INTERVAL_MS (1,000 ms); they are not configurable through environment variables. The first observation is immediate, so the default bound performs at most four one-second sleeps plus the five gateway/container queries. The bounds are intentionally fixed. Allowing environment input to weaken or extend the cleanup proof would make a security gate deployment-dependent. On a host that cannot prove absence within the bound, onboarding fails closed; select compatibility from the outset with NEMOCLAW_DOCKER_GPU_PATCH=1 instead of weakening the handoff proof. If deletion or container cleanup cannot be proven safe, onboarding stops before the retry and prints manual cleanup guidance. Image build, upload, TLS, provider, policy, dashboard, and inference failures stay on their existing error paths and do not trigger the GPU compatibility fallback. Set NEMOCLAW_DOCKER_GPU_PATCH=1 to use only the compatibility path for diagnostics or older host compatibility. Other legacy nonzero values keep that behavior through the v0.0.x release line and will be removed in v0.1.0.

Docker Desktop WSL compatibility route

Automatic GPU onboarding uses the compatibility path directly; it does not make a native attempt first. The path creates the sandbox and then recreates the OpenShell-managed Docker container with NVIDIA GPU flags. NEMOCLAW_DOCKER_GPU_PATCH=0 is ignored because this runtime requires the compatibility patch for GPU passthrough, and onboarding logs a warning when it is set. To skip GPU passthrough entirely, rerun with --no-gpu or set NEMOCLAW_SANDBOX_GPU=0.

Jetson and Tegra compatibility default

Automatic GPU onboarding uses the compatibility path directly; it does not make a native attempt first. The path recreates the OpenShell-managed Docker container with NVIDIA GPU flags and propagates eligible host group IDs for the supported Jetson GPU device nodes. Use NEMOCLAW_DOCKER_GPU_PATCH=0 only for troubleshooting because it bypasses that group propagation and CUDA may not initialize.

Common compatibility-path recovery

After compatibility recreation starts, onboarding keeps the pre-patch container as a rollback backup until the replacement passes the Ready, direct GPU, and applicable local-inference checks. If a later check fails, onboarding prints failure diagnostics and attempts to restore the pre-patch container before it exits. When rollback succeeds, the pre-patch sandbox remains available. If the failed replacement may remain and NemoClaw retains its validated exact container ID, it prints only an exact-container docker rm -f command. If replacement cleanup cannot be confirmed without a validated exact ID, onboarding reports cleanup as unknown and prints no deletion command. When rollback fails, onboarding reports that the sandbox and container state is uncertain and prints no deletion command. A diagnostic bundle captured before rollback records cleanup as pending and contains no deletion command. Inspect the diagnostics, the sandbox, and its labeled Docker containers before removing anything.

Starting with NemoClaw v0.0.43, the standard installer handles the `/proc//task//comm` permission case during this patch path. If an older release fails direct GPU proof with that path and `Permission denied`, upgrade NemoClaw and rerun onboarding.

When inspection confirms that the failed sandbox remains, delete it with a command such as:

openshell sandbox delete <sandbox-name>

Fix the NVIDIA Container Toolkit or CDI configuration reported in the diagnostics. Run the deletion command only after confirming that the pre-patch sandbox was not restored, then rerun onboarding. If you do not need GPU access inside the sandbox, rerun with --no-sandbox-gpu.

If sandbox creation fails with CDI device injection failed: unresolvable CDI devices nvidia.com/gpu=all, the OpenShell gateway tried docker create --device nvidia.com/gpu=all and Docker could not resolve the CDI spec. This injection happens inside the gateway, so NEMOCLAW_DOCKER_GPU_PATCH=0 does not bypass it. Rerun with --no-gpu, or set NEMOCLAW_SANDBOX_GPU=0 and resume onboarding.

If onboarding reports OpenShell supervisor did not reconnect to the GPU-enabled container. even though the diagnostic bundle shows the patched container is running and healthy, the supervisor-reconnect wait is treating a transient Error phase (reported while the OpenShell host re-registers the new container) as fatal. The reconnect wait debounces consecutive Error-phase polls before fast-failing, defaulting to fifteen consecutive polls of about 30 seconds in total. Increase the debounce window with NEMOCLAW_DOCKER_GPU_SUPERVISOR_RECONNECT_ERROR_DEBOUNCE if your host needs more time to re-register the patched container, for example slow WSL2 + Docker Desktop setups. Set it to a higher integer such as 30 (about 60 seconds) and rerun onboarding; the value is clamped to a minimum of 1. If reconnect still fails after the GPU patch, NemoClaw attempts to restore the pre-patch CPU container before exiting. When rollback succeeds, the output says the pre-patch sandbox was restored. When rollback fails, the error says rollback failed and the pre-patch container was not restored, so inspect Docker state before retrying.

pip install fails with a system-packages error

Recent Ubuntu releases (including DGX Spark's Ubuntu 24.04) mark the system Python install as externally managed, so pip install without a virtual environment fails. Use a venv instead. Avoid --break-system-packages unless you understand the risk, since it can break host tooling.

python3 -m venv ~/.venvs/nemoclaw
source ~/.venvs/nemoclaw/bin/activate
pip install ...

Port 3000 conflict with AI Workbench

NVIDIA AI Workbench's Traefik proxy binds ports 3000 and 10000. If you run other services on Spark that expect port 3000, bind them to a different port.

Windows Subsystem for Linux

For environment setup steps, refer to Windows Prerequisites.

wsl --install --no-distribution returns Forbidden (403)

Check your network connectivity. If you are behind a VPN, try reconnecting or switching to a different network. If your network or Windows image blocks the online WSL installer, install WSL manually with Microsoft's offline install guidance. Download the latest official WSL .msi package from the Microsoft WSL releases page, choose the matching .x64.msi or .arm64.msi, install it, reboot if Windows requests it, then rerun wsl --status.

Bootstrap says "Windows Subsystem for Linux is not fully installed"

The bootstrap script checks wsl --status before it installs or opens Ubuntu. If Windows reports that the WSL runtime is not installed, the script attempts wsl --install --no-distribution automatically. If the repair command succeeds and WSL reports that the changes require a reboot, reboot and let the bootstrap resume after sign-in. If the repair command succeeds but WSL still cannot be verified and the output does not request a reboot, follow the printed repair guidance instead of rebooting by default. If the repair command fails, follow the printed repair steps. If the repair command returns Forbidden (403) or remains blocked, install WSL manually with Microsoft's offline install guidance. Download the latest official WSL .msi package from the Microsoft WSL releases page, choose the matching .x64.msi or .arm64.msi, install it, reboot if Windows requests it, then rerun the bootstrap script. Use the same repair flow if the bootstrap says "Windows Subsystem for Linux could not be verified" and reports a nonzero wsl --status exit code.

Bootstrap says "Windows reports that WSL 2 cannot start yet"

The bootstrap script attempts wsl --install --no-distribution automatically when wsl --status reports that WSL 2 cannot start. If the repair command succeeds, reboot when prompted and let the bootstrap resume after sign-in. If the message persists after repair and reboot, enable virtualization in firmware and confirm that the Virtual Machine Platform optional component is enabled. Manual WSL installation only helps when the WSL runtime is missing or the online installer is blocked.

wsl -d Ubuntu says "There is no distribution with the supplied name"

The Ubuntu package was installed with --no-launch but never registered, or Windows finished the install command before the distribution appeared in wsl -l. When this happens during the NemoClaw bootstrap, the script prints a sanitized WSL install output block. PowerShell transcript headers, footers, temporary transcript paths, and status-file paths are redacted before display so you can paste the useful WSL output into a bug report with less local machine metadata.

If the sanitized output says a reboot is required, reboot and rerun the bootstrap. If it does not request a reboot, register the distro manually or reinstall without --no-launch:

wsl --unregister Ubuntu
wsl --install -d Ubuntu

Bootstrap says a Docker executable "is not signed by a trusted publisher"

The bootstrap script runs elevated, so before it launches Docker Desktop.exe or uses docker.exe, it checks the resolved executable's Authenticode signature and refuses to run one that is not validly signed by Docker. The script accepts Docker Inc as the certificate subject common name. If Docker changes the signer identity, the script refuses the executable until maintainers verify the signer on an official Docker download and update the allowlist. For a current-user installation, the administrator child completes the system changes and returns to the original non-elevated PowerShell process before the script starts Docker Desktop or uses its CLI. If you started the script from an elevated PowerShell window, rerun it from a normal PowerShell window so it can use the current-user installation without administrator privileges. Reinstall Docker Desktop from docker.com or winget install --id Docker.DockerDesktop, then rerun the bootstrap script. If reinstalling does not clear the warning, treat the existing executable as untrusted and do not run it manually either.

The script continues after this warning instead of stopping, so the docker info fails inside WSL symptom below can appear a few minutes later even though the real cause is the untrusted executable, not WSL integration.

docker info fails inside WSL

Confirm that Docker Desktop is running and that WSL integration is enabled for Ubuntu (Settings > Resources > WSL integration). Then restart WSL:

wsl --shutdown
wsl -d Ubuntu
docker info

Windows-host Ollama is installed but not shown during onboarding

When NemoClaw runs inside WSL, it checks both the Windows-host Ollama HTTP endpoint and the Windows ollama.exe process. If Ollama is installed but the daemon is not reachable through host.docker.internal:11434, the wizard should still offer a start or restart action.

If the Windows-host option does not appear, confirm that PowerShell interop is enabled in WSL and that Windows can locate Ollama:

powershell.exe -NoProfile -Command "Get-Process ollama -ErrorAction SilentlyContinue"

If the process is missing, start Ollama from Windows and rerun onboarding. If the process exists but the endpoint is unreachable, use the restart action when the wizard offers it, or restart Ollama from Windows with OLLAMA_HOST=0.0.0.0:11434.

Ollama inference fails or hangs in WSL

Ollama configures context length based on your hardware.

On some GPUs (for example RTX 3500), the default context length is not sufficient for OpenClaw. During onboarding, NemoClaw raises loaded-model context lengths below 16384 to 16384 when NEMOCLAW_CONTEXT_WINDOW is unset. Set the variable manually when you need a different value or when you run Ollama outside the managed onboarding path. Force a larger context length:

pkill -f 'ollama serve'
OLLAMA_CONTEXT_LENGTH=16384 ollama serve

Hermes requires at least 64000 tokens. During onboarding, NemoClaw verifies the loaded model's actual context_length through Ollama's /api/ps endpoint. Resumed onboarding and sandbox rebuilds warm the exact recorded Ollama model and repeat this check before reusing its route. Resume stops when that recorded model is missing, Ollama is unreachable, model warm-up fails, or the runtime context cannot be verified. If you set NEMOCLAW_CONTEXT_WINDOW above 64000, the loaded model must provide at least that larger value. If the runtime value is too small, missing, or malformed, onboarding stops before sandbox creation and asks you to restart the host daemon with the required context length. NEMOCLAW_CONTEXT_WINDOW controls Hermes prompt budgeting; it does not change the Ollama daemon or bypass this runtime check. Force a larger context length:

pkill -f 'ollama serve'
OLLAMA_CONTEXT_LENGTH=64000 ollama serve

Verify that Ollama inference works:

echo "Hello" | ollama run <model-id>

Replace <model-id> with the model you selected during onboarding (for example qwen3.5:4b).

If ollama serve fails with Error: listen tcp 127.0.0.1:11434: bind: address already in use, check whether Ollama is configured for automatic startup:

sudo systemctl status ollama

If it is active, stop it first, then start with the custom context length:

sudo systemctl stop ollama
OLLAMA_CONTEXT_LENGTH=16384 ollama serve

sudo systemctl stop ollama
OLLAMA_CONTEXT_LENGTH=64000 ollama serve

For additional troubleshooting, refer to the Windows Setup page. For first-time OpenClaw setup, refer to the Quickstart. For first-time Hermes setup, refer to Quickstart with Hermes.

Podman

Podman is not a tested runtime. OpenShell officially documents Docker-based runtimes only. If you encounter issues with Podman, switch to a tested runtime (Docker Engine, Docker Desktop, or Colima) and rerun onboarding.

The portable experimental profile uses the docker command to drive rootless Podman. Before you run this profile, make sure a Docker-compatible CLI is available on PATH. On a Podman-only host, install the podman-docker shim for your distribution:

# Debian or Ubuntu
sudo apt install podman-docker

# Fedora
sudo dnf install podman-docker

Then rerun portable onboarding:

$$nemoclaw onboard --experimental-profile portable

Portable Podman Readiness Fails

Portable commands use the current user's rootless Podman socket authority recorded in NemoClaw state. They ignore ambient Docker and Podman runtime selectors, including named connections. Do not export another DOCKER_HOST, DOCKER_CONTEXT, CONTAINER_HOST, or CONTAINER_CONNECTION to bypass a readiness failure.

When podman.service reports inactive and the recorded socket exists, NemoClaw first makes one 10-second API request through the guarded recorded authority. A valid server version classifies the endpoint as warm and avoids starting another socket service. A missing socket or a response without a valid server version enters bounded cold activation. Any socket authority change during this precheck fails at the socket authority stage and is not eligible for inode requalification.

During cold activation, the first API probe can cause systemd to replace the socket inode. NemoClaw requalifies one such replacement and repeats the probe only when the socket path, device, mode, owner, and complete directory authority remain unchanged. Any other authority change or a second inode replacement fails at the socket authority stage.

A portable readiness failure identifies the stage that did not complete:

Stage	Meaning	Recovery
Socket authority	The recorded socket is unsafe, another authority field changed, or its inode was replaced more than once during cold activation.	Stop and inspect the reported path. Restore the recorded current-user runtime instead of selecting another endpoint.
Service activation	The current user's systemd manager could not activate the Podman socket-backed service, or activation did not create the recorded socket within the startup period.	Inspect podman.socket and podman.service with the commands below. Correct the reported user-unit failure, then rerun the NemoClaw command.
Startup API health	The service was activated, but the recorded endpoint did not return a real Podman API response within the startup period.	Inspect the user-unit logs and run the explicit API request below against the reported socket path. Raise NEMOCLAW_PORTABLE_PODMAN_STARTUP_TIMEOUT_MS only when valid cold activation needs more than 60,000 ms.
Steady-state API health	An endpoint that completed activation did not answer the later shorter health check.	Inspect host load and the user-unit logs, then rerun the NemoClaw command.

Inspect the current user's units without changing them:

systemctl --user status podman.socket podman.service --no-pager
journalctl --user -u podman.socket -u podman.service --since -10m --no-pager

If the units need activation, restart the active service when present and start the socket for the current user session:

systemctl --user try-restart podman.service
systemctl --user start podman.socket

These commands do not enable the socket for later user sessions.

Use the exact socket path from the NemoClaw failure to require a real server response:

podman --remote \
  --url unix://<recorded-socket-path> \
  version \
  --format 'Server Version: {{.Server.Version}}'

Continue only when the command exits with status 0 and prints a nonempty server version. The request and the readiness report contain no credentials. Rerun the original NemoClaw command without exporting a Docker or Podman runtime selector.

If valid cold activation needs a larger budget, set an integer from 15000 through 300000 milliseconds:

export NEMOCLAW_PORTABLE_PODMAN_STARTUP_TIMEOUT_MS=120000
$$nemoclaw <original-arguments>

The default cold-start budget is 60,000 ms. The later steady-state API deadline is fixed at 10,000 ms and does not use this setting. A successful cold path uses the cold timing label and reports activation, API, and total time in milliseconds. A successful warm path uses the warm timing label and reports steady-state API and total time in milliseconds.

Portable Host Gateway Is Unreachable

The portable experimental profile maps host.openshell.internal to the OpenShell Podman host gateway. Do not apply the Docker bridge UFW command when portable onboarding reports this route as unreachable.

Use the same procedure when onboarding reports that the Podman service is unreachable for the portable gateway probe.

Portable onboarding reports output like this:

✗ Sandbox containers cannot reach the gateway at host.openshell.internal:8080.
  The probe mapped host.openshell.internal to the OpenShell Podman host gateway.

If podman.service is active, restart it. Then start the user-scoped Podman socket for the current user session:

systemctl --user try-restart podman.service
systemctl --user start podman.socket

The first command does not start an inactive service. The second command starts the current user's Podman API socket without enabling it for later user sessions. These commands affect only the current user's Podman units. They do not read or write credentials.

Verify that the socket is active:

systemctl --user is-active podman.socket

Expected output:

active

An active socket alone does not establish API health. Run the explicit Podman API request from Portable Podman Readiness Fails before you rerun onboarding.

Then rerun portable onboarding:

$$nemoclaw onboard --experimental-profile portable

If the failed run has a checkpoint schema 4 resumable session, resume it without exporting Docker or Podman runtime selectors:

$$nemoclaw onboard --resume

Continue only when onboarding no longer reports that the Podman service or OpenShell Podman host gateway is unreachable.

Hermes

The issues below are common problems you may encounter when running Hermes through nemohermes. For setup, refer to Quickstart with Hermes.

Hermes dashboard config did not converge

nemohermes inference set updates the OpenShell route, registry, and /sandbox/.hermes/config.yaml before it refreshes the separate dashboard profile. If the dashboard profile exists but NemoClaw cannot confirm that /sandbox/.hermes/profiles/dashboard-home/config.yaml was updated, the command exits nonzero without printing Inference route synced. The committed route and main Hermes config are not rolled back.

Restart the sandbox so startup mirrors the committed model route into the dashboard profile:

nemohermes <name> stop
nemohermes <name> start

Then run nemohermes inference get and verify Dashboard Chat uses the selected model. If the command succeeds because the dashboard profile is missing, the dashboard is disabled and no dashboard recovery is required.

Shields Reports Drift for the Hermes Configuration Root

The Hermes configuration root holds the agent's top-level runtime state, not only its configuration. The configuration root stores auth.json, the drain request, and temporary files for atomic gateway_state.json and gateway.pid replacement. Lockdown moves that directory to root:sandbox mode 3770, which preserves the set-group-ID and sticky bits. The gateway can still manage its runtime files. The sticky bit prevents the sandbox identity from unlinking or renaming sealed root-owned configuration. Run the following command to inspect a locked root:

$$nemoclaw <name> exec -- stat -c '%a %U:%G' /sandbox/.hermes

Expected output:

3770 root:sandbox

A sandbox locked by an older release carries a 755 root:root root instead. The gateway cannot write its runtime state there. $$nemoclaw <name> shields status reports the stale posture as drift. Before restarting the gateway, repair the posture:

$$nemoclaw <name> shields up

If the command refuses the repair, follow its recovery guidance and do not restart the gateway. Otherwise, verify the repaired posture:

$$nemoclaw <name> shields status
$$nemoclaw <name> exec -- stat -c '%a %U:%G' /sandbox/.hermes

Continue only when status no longer reports drift and stat prints the expected output above. Then restart the gateway:

$$nemoclaw <name> gateway restart

The restart must exit zero and report that the health check passed. If restart or recovery already reports relaunch quarantined, repairing the directory cannot clear the supervisor quarantine. Follow Restart or recovery reports relaunch quarantined to rebuild the sandbox. The sealed files are unchanged by the repair and stay 444 root:root.

Hermes restart reports config hash mismatch

A Hermes restart reports config hash mismatch when a strict root-owned hash is available and /sandbox/.hermes/config.yaml or /sandbox/.hermes/.env does not match it. The direct root-entrypoint supervisor always uses the strict hash at /etc/nemoclaw/hermes.config-hash. The OpenShell-managed controller uses that hash when both config inputs are root-owned and locked. Mutable config in the managed topology has no durable root-owned hash anchor, so restart retains the same trust and time-of-check/time-of-use limits as managed cold start and cannot use this error to prove direct drift. Both controllers validate the secret boundary and supervisor runtime environment before they stop the tracked gateway. They do not recompute a trusted strict hash to adopt direct edits made inside the sandbox.

For intended changes, use supported host commands such as nemohermes <name> config set and nemohermes inference set so NemoClaw updates the config and its strict and compatibility hashes together. Do not edit either hash file manually.

If direct edits have already caused a mismatch, restore the original config and environment files or rebuild from the registered configuration:

nemohermes <name> rebuild --yes

If the command instead reports secret-boundary refusal, inspect /sandbox/.hermes/.env for raw secret-shaped values. Replace them through the supported credential flow so the file contains openshell:resolve:env:<name> placeholders, then run nemohermes <name> recover. The Hermes entrypoint supervisor remains responsible for the gateway, dashboard, internal API relay, dashboard relay, and gateway log stream throughout recovery. In the OpenShell-managed topology, that nonroot supervisor repairs failed auxiliaries continuously, recovers a gateway after four consecutive failed listener or HTTP health checks, and quarantines relaunch after five exits within 60 seconds until the sandbox is recreated. The host only repairs the host-side OpenShell forwards after the supervised processes pass health checks.

Restart or recovery reports relaunch quarantined

In the OpenShell-managed topology the strict root-owned hash is not a trust anchor for mutable config, so a direct edit of /sandbox/.hermes/config.yaml or /sandbox/.hermes/.env is not refused by the host controller. The in-sandbox supervisor still refuses to start a gateway on configuration it cannot match to the persisted managed state, and it stops attempting relaunch once that refusal or repeated gateway exits exhaust its crash budget. $$nemoclaw <name> gateway restart, $$nemoclaw <name> recover, and $$nemoclaw <name> connect then report the relaunch quarantined failure layer.

The refusal is deterministic, so retrying any of those commands cannot clear it. Restore the registered configuration and refresh its integrity metadata in one transaction:

nemohermes <name> rebuild --yes

After the rebuild, make the intended change through a supported command such as $$nemoclaw <name> config set or $$nemoclaw inference set, which update the configuration and its hashes together.

Port 8642 in a browser shows a blank page or Cannot GET /

nemohermes onboard forwards the sandbox's API port, which is 8642 when no other sandbox or host listener already holds it. Hermes serves an OpenAI-compatible API at that port, not a chat dashboard. A browser visit to http://127.0.0.1:8642/ (or any non-API path) returns nothing renderable.

Confirm the agent is healthy with the API health endpoint instead:

curl -sf http://127.0.0.1:8642/health

Expected output:

{"status":"ok","platform":"hermes-agent"}

Point an OpenAI-compatible client at http://127.0.0.1:8642/v1 for chat completions. For terminal use, run nemohermes launch <name>.

docker port shows no mapping for 8642 even though forwarding works

OpenShell port forwards are host-side relays managed by the OpenShell gateway process, not Docker -p publish mappings on the sandbox container. docker port openshell-hermes-<id> reflects only Docker-published ports, so it returns nothing for OpenShell-managed forwards even when the host bind is live.

Use OpenShell's own view as the supported acceptance signal:

openshell forward list                       # shows the host bind for each forwarded port
curl -sf http://127.0.0.1:8642/health        # confirms the relayed endpoint answers

If openshell forward list does not show the sandbox's API port, run nemohermes <name> connect --probe-only (or nemohermes <name> recover) to ask the recovery path to re-establish every manifest-declared agent forward port that has gone missing. Recovery targets each sandbox's own ports. A second Hermes sandbox on the same host receives the next free API port, so check which sandbox owns each row before assuming a missing 8642 row belongs to the sandbox you are debugging. A Hermes sandbox onboarded before the API port became per-sandbox carries no allocated port and keeps 8642. A Hermes sandbox created after that change receives its own port during onboarding, so a second sandbox needs no further action. To move a sandbox that predates the change onto its own port, set NEMOCLAW_HERMES_API_PORT=<N> and rerun onboarding with --recreate-sandbox. Set <N> to a free port from 8642 through 8652. Onboarding rejects a value outside that range. It also rejects an in-range value that another sandbox or host listener already holds. A recreate keeps the sandbox's registry entry, so --recreate-sandbox without the variable keeps the recorded port.

Install reports Could not restore the Hermes forward

The installer reads the sandbox's recorded API port from the sandbox registry before it restores the API forward. It exits with that message when any of these conditions is true:

• The node binary is missing.
• The registry file is missing, does not parse, or records no entry for the sandbox.
• The recorded port is not an integer from 8642 through 8652.

A sandbox registered before the API port became per-sandbox records no port, and the installer uses 8642 for it. Install node, or rerun nemohermes onboard to register the sandbox again. Then run nemohermes <name> recover to re-establish the forward.

nemohermes reports Sandbox 'X' already exists as OpenClaw

Each sandbox name maps to exactly one agent type. If a sandbox named X was created with the default OpenClaw agent, a later nemohermes onboard for the same name exits with:

Sandbox 'X' already exists as OpenClaw.
nemohermes is onboarding Hermes for this sandbox name.
Side-by-side agents are supported, but each sandbox name has one agent type.

Pick a distinct sandbox name (the Hermes default is hermes; a common pattern is my-hermes) so Hermes and OpenClaw sandboxes can coexist on the same host. To convert an existing sandbox to Hermes instead, destroy and re-onboard:

$$nemoclaw <name> destroy
NEMOCLAW_AGENT=hermes nemohermes onboard

nemohermes: command not found immediately after install

nemohermes is a thin shim installed alongside $$nemoclaw that pre-selects the Hermes agent. The installer drops the shim in the same directory as $$nemoclaw; if $$nemoclaw is on PATH but nemohermes is not, the shim symlink was skipped.

Verify the install:

command -v nemoclaw
command -v nemohermes

If only $$nemoclaw resolves, re-run the installer with NEMOCLAW_AGENT=hermes set so the shim is published:

export NEMOCLAW_AGENT=hermes
curl -fsSL https://www.nvidia.com/nemoclaw.sh | bash

Equivalently, every nemohermes <cmd> invocation is NEMOCLAW_AGENT=hermes nemoclaw <cmd>.

Choosing between OAuth and API key for the Hermes Provider

The Hermes Provider supports two authentication paths during onboarding. Pick OAuth when you have a Nous Portal account and an interactive terminal; pick API key when you have a long-lived NOUS_API_KEY and want a non-interactive flow.

Set the method explicitly so the wizard skips the prompt:

# OAuth (default; interactive)
export NEMOCLAW_HERMES_AUTH_METHOD=oauth
nemohermes onboard

# API key (non-interactive)
export NEMOCLAW_HERMES_AUTH_METHOD=api-key
export NOUS_API_KEY=nous_...
nemohermes onboard --non-interactive

NEMOCLAW_HERMES_AUTH_METHOD accepts oauth, nous-portal-oauth, api-key, and nous-api-key. The NEMOCLAW_HERMES_AUTH and NEMOCLAW_NOUS_AUTH_METHOD variables are back-compatible aliases.

If OAuth is selected and onboarding cannot open the host's default browser (a headless host or SSH session), the device-code prompt still prints the verification URL and user code to the terminal. Copy them to a browser on any other machine to complete the flow.

API client returns 401 Unauthorized against port 8642

Hermes uses bearer-token header authentication for client requests, not an OpenClaw-style URL fragment. A request without an Authorization: Bearer <token> header (or with an OpenClaw #token= fragment appended to the URL) is rejected with 401.

Configure your OpenAI-compatible client to pass the Hermes API key in the Authorization header. Stored credentials (including NOUS_API_KEY and OPENAI_API_KEY) are listed by:

nemohermes credentials list

Reset a specific provider's credentials with nemohermes credentials reset <provider> and re-onboard if the stored value is wrong.

Brave Search is unsupported under Hermes

Hermes does not have a NemoClaw Brave Search backend. Adding the brave preset to a Hermes sandbox opens Brave's endpoints but does not configure Hermes to use the credential. Use Tavily Search through NemoClaw onboarding instead.

NEMOCLAW_WEB_SEARCH_PROVIDER=tavily \
TAVILY_API_KEY=<your-tavily-key> \
  nemohermes onboard --recreate-sandbox

NemoClaw writes web.backend: tavily, applies the tavily policy preset, and configures request-body credential rewriting for Hermes. If the same onboarding run selected the Nous-managed web gateway, Tavily replaces nous-web while selected Nous image, audio, browser, and code tools remain enabled.

Re-onboarding asks every messaging prompt again

nemohermes onboard --resume against a Hermes sandbox that was originally onboarded with Telegram, Discord, and Slack credentials re-prompts for each channel's bot token and per-channel settings rather than reusing the stored values. This is tracked in #3581. For unattended re-onboards, export the messaging env vars first so the wizard skips the prompts:

export TELEGRAM_BOT_TOKEN=...
export DISCORD_BOT_TOKEN=...
export SLACK_BOT_TOKEN=...
export SLACK_APP_TOKEN=...
nemohermes onboard --resume --non-interactive