Skip to content
Merged
Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
1ac29a5
fix(portable): separate Podman activation readiness
senthilr-nv Aug 14, 2026
fbfaab1
merge(portable): reconcile readiness with current main
senthilr-nv Aug 14, 2026
1a7cc83
fix(onboard): keep authority handoff net-neutral
senthilr-nv Aug 14, 2026
5183e5d
test(portable): keep readiness cases linear
senthilr-nv Aug 14, 2026
bb8b8b5
fix(portable): preserve activation readiness
cv Aug 14, 2026
773c5da
fix(portable): bound socket requalification
rsliter Aug 14, 2026
c674f5c
refactor(onboard): remove unused sandbox type
rsliter Aug 14, 2026
139d519
test(portable): allow slower lock coverage
rsliter Aug 14, 2026
81816dc
fix(portable): reuse healthy pinned API
rsliter Aug 15, 2026
d65dd2e
docs(portable): explain warm API reuse
rsliter Aug 15, 2026
773c5df
merge: resolve conflicts with main
github-actions[bot] Aug 15, 2026
db48d0b
fix(portable): bind lifecycle mutations to Podman
cv Aug 15, 2026
cccf7ee
test(portable): prove cold readiness before gateway
cv Aug 15, 2026
607212c
test(portable): stop proof service before readiness
cv Aug 15, 2026
6e7eac4
fix(e2e): use supported podman cleanup timeout
cv Aug 15, 2026
a47121f
test(onboard): provide portable runtime authority
cv Aug 15, 2026
5834343
merge: refresh PR branch with main
senthilr-nv Aug 15, 2026
ea1af64
test(doctor): cover portable runtime readiness
cv Aug 15, 2026
0582c76
merge: integrate PR branch refresh
cv Aug 15, 2026
e38bcd2
test(portable): freeze readiness budget clock
prekshivyas Aug 15, 2026
82b6c43
docs(portable): clarify socket authority recovery
senthilr-nv Aug 15, 2026
8740968
merge: incorporate current main
cv Aug 15, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions docs/reference/commands.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -874,6 +874,15 @@ After all portable TCP probe attempts fail, onboarding prints commands for the u

Onboarding prints the same commands when the portable probe cannot reach the user-scoped Podman service.
The printed rerun command keeps the portable experimental profile selected.
Portable commands reconstruct the current user's rootless Podman socket authority from NemoClaw state before they use the Docker-compatible API.
They do not select an endpoint from ambient Docker or Podman runtime variables or named connections.
When the user-scoped socket-backed service needs activation, NemoClaw activates it and waits through a bounded startup period for a real Podman API response.
Only a successful response permits the shorter steady-state API health deadline.
Onboarding and portable sandbox lifecycle commands use this same readiness contract.
Failures identify socket authority, service activation, startup API health, or steady-state API health without reporting credentials.
NemoClaw does not fall back to Docker or report an absent or unreachable endpoint as healthy.
A successful cold path uses the `cold` timing label and reports activation, API, and total time in milliseconds.
A successful warm path uses the `warm` timing label and reports steady-state API and total time in milliseconds.
To tune the existing-gateway HTTP health poll, use `NEMOCLAW_REUSE_HEALTH_POLL_COUNT` (default `6`) and `NEMOCLAW_REUSE_HEALTH_POLL_INTERVAL` (default `5` seconds).
The poll count is clamped to a minimum of `1` so the health probe always runs at least once, and the interval is clamped to a minimum of `0` (no sleep between attempts).

Expand Down Expand Up @@ -4958,6 +4967,7 @@ Defaults are sized for typical hardware; override only if you see false-positive
| Variable | Default | Effect |
|----------|---------|--------|
| `NEMOCLAW_MCP_PROVIDER_SYNC_TIMEOUT_SECONDS` | `30` | Maximum time to wait for an OpenShell MCP provider credential revision to become active or fully revoked inside the sandbox. Integer seconds; raise only when provider synchronization is unusually slow. |
| `NEMOCLAW_PORTABLE_PODMAN_STARTUP_TIMEOUT_MS` | `60000` | Maximum cold-start time for portable rootless Podman socket activation and the first real API response. Set an integer from `15000` through `300000` milliseconds. This setting does not change the fixed 10,000 ms steady-state API deadline. |
| `NEMOCLAW_SANDBOX_EXEC_TIMEOUT_MS` | per call site (typically `15000`) | Overrides the default timeout for `openshell sandbox exec` calls issued by recovery and lifecycle helpers. Integer milliseconds; non-positive or non-numeric values fall back to the per-call-site default. |
| `NEMOCLAW_STATUS_PROBE_TIMEOUT_MS` | built-in default | Overrides the timeout for the OpenShell status probe used by `$$nemoclaw <name> status`. Integer milliseconds; non-positive or non-numeric values fall back to the default. |
| `NEMOCLAW_WSL_GPU_PROOF_TIMEOUT_MS` | `180000` | Maximum time for the bounded Docker CUDA workload on an eligible ARM64 Linux host. A positive finite number of milliseconds overrides the default. Invalid, infinite, zero, and negative values use the default. |
Expand Down
57 changes: 57 additions & 0 deletions docs/reference/troubleshooting.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -3329,6 +3329,60 @@ Then rerun portable onboarding:
$$nemoclaw onboard --experimental-profile portable
```

### Portable Podman Readiness Fails

Portable commands use the current user's rootless Podman socket authority recorded in NemoClaw state.
They ignore ambient Docker and Podman runtime selectors, including named connections.
Do not export another `DOCKER_HOST`, `DOCKER_CONTEXT`, `CONTAINER_HOST`, or `CONTAINER_CONNECTION` to bypass a readiness failure.

A portable readiness failure identifies the stage that did not complete:

| Stage | Meaning | Recovery |
| --- | --- | --- |
| Socket authority | The recorded socket is unsafe, is replaced, or does not match the current user's recorded runtime authority. | Stop and inspect the reported path. Restore the recorded current-user runtime instead of selecting another endpoint. |
| Service activation | The current user's systemd manager could not activate the Podman socket-backed service, or activation did not create the recorded socket within the startup period. | Inspect `podman.socket` and `podman.service` with the commands below. Correct the reported user-unit failure, then rerun the NemoClaw command. |
| Startup API health | The service was activated, but the recorded endpoint did not return a real Podman API response within the startup period. | Inspect the user-unit logs and run the explicit API request below against the reported socket path. Raise `NEMOCLAW_PORTABLE_PODMAN_STARTUP_TIMEOUT_MS` only when valid cold activation needs more than 60,000 ms. |
| Steady-state API health | An endpoint that completed activation did not answer the later shorter health check. | Inspect host load and the user-unit logs, then rerun the NemoClaw command. |

Inspect the current user's units without changing them:

```bash
systemctl --user status podman.socket podman.service --no-pager
journalctl --user -u podman.socket -u podman.service --since -10m --no-pager
```

If the units need activation, restart the active service when present and enable the socket:

```bash
systemctl --user try-restart podman.service
systemctl --user enable --now podman.socket
```

Use the exact socket path from the NemoClaw failure to require a real server response:

```bash
podman --remote \
--url unix://<recorded-socket-path> \
version \
--format 'Server Version: {{.Server.Version}}'
```

Continue only when the command exits with status `0` and prints a nonempty server version.
The request and the readiness report contain no credentials.
Rerun the original NemoClaw command without exporting a Docker or Podman runtime selector.

If valid cold activation needs a larger budget, set an integer from `15000` through `300000` milliseconds:

```bash
export NEMOCLAW_PORTABLE_PODMAN_STARTUP_TIMEOUT_MS=120000
$$nemoclaw <original-arguments>
```

The default cold-start budget is 60,000 ms.
The later steady-state API deadline is fixed at 10,000 ms and does not use this setting.
A successful cold path uses the `cold` timing label and reports activation, API, and total time in milliseconds.
A successful warm path uses the `warm` timing label and reports steady-state API and total time in milliseconds.

### Portable Host Gateway Is Unreachable

The portable experimental profile maps `host.openshell.internal` to the OpenShell Podman host gateway.
Expand Down Expand Up @@ -3368,6 +3422,9 @@ Expected output:
active
```

An active socket alone does not establish API health.
Run the explicit Podman API request from [Portable Podman Readiness Fails](#portable-podman-readiness-fails) before you rerun onboarding.

Then rerun portable onboarding:

```bash
Expand Down
23 changes: 23 additions & 0 deletions src/lib/actions/sandbox/doctor-lifecycle-registration.ts
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ import {
collectLifecycleRegistrationIssues,
type LifecycleRegistrationIssue,
} from "../../domain/lifecycle-registration";
import { inspectPortableRuntimeReceiptReadiness } from "../../onboard/experimental/portable-runtime-receipt-readiness";
import type { SandboxEntry } from "../../state/registry";
import type { DoctorCheck } from "./doctor-report";

Expand All @@ -23,6 +24,28 @@ function formatFieldList(
.join(", ");
}

export function buildPortableRuntimeCheck(sandboxName: string): DoctorCheck | null {
const portable = inspectPortableRuntimeReceiptReadiness(sandboxName);
if (!portable) return null;
const recordedSocket = !portable.ok && portable.socketPath
? ` Recorded socket: ${portable.socketPath}.`
: "";
return portable.ok
? {
group: "Host",
label: "Portable Podman API",
status: "ok",
detail: `server ${portable.serverVersion}; ${portable.timing.mode}; activation ${String(portable.timing.activationMs)} ms; API ${String(portable.timing.apiMs)} ms; total ${String(portable.timing.totalMs)} ms`,
}
: {
group: "Host",
label: "Portable Podman API",
status: "fail",
detail: `${portable.stage}: ${portable.detail}${recordedSocket}`,
hint: "repair the recorded current-user Podman endpoint, then retry",
};
}

export function buildLifecycleRegistrationCheck(
sandboxName: string,
entry: SandboxEntry,
Expand Down
41 changes: 26 additions & 15 deletions src/lib/actions/sandbox/doctor.ts
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,10 @@ import {
type DoctorInferenceRoute,
resolveDoctorReasoningEffort,
} from "./doctor-inference";
import { buildLifecycleRegistrationCheck } from "./doctor-lifecycle-registration";
import {
buildLifecycleRegistrationCheck,
buildPortableRuntimeCheck,
} from "./doctor-lifecycle-registration";
import { collectMessagingDoctorChecks } from "./doctor-messaging";
import {
buildDoctorReport,
Expand Down Expand Up @@ -136,36 +139,44 @@ function cliBuildCheck(): DoctorCheck {
};
}

function collectHostChecks(sb: SandboxEntry | null | undefined): {
checks: DoctorCheck[];
openshellBin: ReturnType<typeof resolveOpenshell>;
} {
const cli = cliBuildCheck();
const openshellBin = resolveOpenshell();
let runtimeCheck: DoctorCheck;
function inspectRuntimeHost(sb: SandboxEntry | null | undefined): DoctorCheck {
const portable = sb ? buildPortableRuntimeCheck(sb.name) : null;
if (portable) return portable;
const recorded = sb?.openshellDriver?.trim();
const provider = recorded
? requireRuntimeProviderBundle(recorded, CURRENT_RUNTIME_PROVIDER_BUNDLES)
: resolveCurrentRuntimeProviderBundle();
return provider.preflightDoctor.inspectHost();
}

function runtimeHostCheck(sb: SandboxEntry | null | undefined): DoctorCheck {
try {
const recorded = sb?.openshellDriver?.trim();
const provider = recorded
? requireRuntimeProviderBundle(recorded, CURRENT_RUNTIME_PROVIDER_BUNDLES)
: resolveCurrentRuntimeProviderBundle();
runtimeCheck = provider.preflightDoctor.inspectHost();
return inspectRuntimeHost(sb);
} catch (error) {
const detail =
error instanceof RuntimeProviderSelectionError
? error.message
: `Runtime provider inspection failed: ${error instanceof Error ? error.message : String(error)}`;
runtimeCheck = {
return {
group: "Host",
label: "Runtime provider",
status: "fail",
detail,
hint: "restore a supported durable runtime provider identity before retrying",
};
}
}

function collectHostChecks(sb: SandboxEntry | null | undefined): {
checks: DoctorCheck[];
openshellBin: ReturnType<typeof resolveOpenshell>;
} {
const cli = cliBuildCheck();
const openshellBin = resolveOpenshell();
return {
checks: [
cli,
runtimeCheck,
runtimeHostCheck(sb),
{
group: "Host",
label: "OpenShell CLI",
Expand Down
42 changes: 42 additions & 0 deletions src/lib/actions/sandbox/gateway-failure-classifier.ts
Original file line number Diff line number Diff line change
Expand Up @@ -9,11 +9,20 @@ import { CLI_NAME } from "../../cli/branding";
import { GATEWAY_PORT } from "../../core/ports";
import { resolveSandboxContainerOwner } from "../../domain/sandbox/container-owner";
import { resolveGatewayPortFromName } from "../../onboard/gateway-binding";
import type { PortablePodmanReadinessResult } from "../../onboard/experimental/portable-runtime-readiness";
import {
inspectPortableRuntimeReceiptReadiness,
type PortableRuntimeReceiptReadinessDeps,
} from "../../onboard/experimental/portable-runtime-receipt-readiness";
import * as registry from "../../state/registry";
import { getSandboxTargetGatewayName } from "./gateway-target";

const DOCKER_TIMEOUT_MS = 3000;
const PORT_PROBE_TIMEOUT_MS = 2000;
const portableRuntimeFailures = new Map<
string,
Extract<PortablePodmanReadinessResult, { ok: false }>
>();

export type GatewayFailureLayer =
| "docker_unreachable"
Expand Down Expand Up @@ -281,8 +290,22 @@ export function isDockerRuntimeDown(
opts?: {
runners?: Pick<GatewayFailureRunners, "dockerInfo">;
getSandbox?: SandboxDriverLookup;
portableLifecycle?: PortableRuntimeReceiptReadinessDeps;
},
): boolean {
const portable = inspectPortableRuntimeReceiptReadiness(sandboxName, opts?.portableLifecycle);
if (portable) {
if (portable.ok) {
portableRuntimeFailures.delete(sandboxName);
console.log(
` Portable Podman readiness: ${portable.timing.mode}; activation ${String(portable.timing.activationMs)} ms; API ${String(portable.timing.apiMs)} ms; total ${String(portable.timing.totalMs)} ms.`,
);
return false;
}
portableRuntimeFailures.set(sandboxName, portable);
return true;
}
portableRuntimeFailures.delete(sandboxName);
const getSandbox = opts?.getSandbox ?? registry.getSandbox;
if (!isDockerBackedSandbox(sandboxName, getSandbox)) return false;
const probe = opts?.runners?.dockerInfo ?? defaultRunners.dockerInfo;
Expand All @@ -302,6 +325,25 @@ export function printDockerRuntimeDownGuidance(
): void {
const writer = opts.writer ?? console.error;
const retryCommand = opts.retryCommand ?? "status";
const portable = portableRuntimeFailures.get(sandboxName);
portableRuntimeFailures.delete(sandboxName);
if (portable) {
writer(` Failure stage: ${portable.stage} — ${portable.detail}`);
if (portable.socketPath) writer(` Recorded socket: ${portable.socketPath}`);
writer(
` Portable Podman readiness (${portable.timing.mode}): activation ${String(portable.timing.activationMs)} ms; API ${String(portable.timing.apiMs)} ms; total ${String(portable.timing.totalMs)} ms.`,
);
writer(
` The receipt-owned Podman endpoint for sandbox '${sandboxName}' is not ready; no Docker or named-connection fallback was used.`,
);
writer(" Recovery:");
writer(
" 1. Check the reported readiness stage and the current user's Podman socket service.",
);
writer(" 2. Confirm the recorded endpoint returns a real Podman server version.");
writer(` 3. Retry: ${CLI_NAME} ${sandboxName} ${retryCommand}`);
return;
}
writer(` ${getLayerHeader("docker_unreachable")}`);
writer(
` The Docker daemon is not reachable, so sandbox '${sandboxName}' cannot be verified or started.`,
Expand Down
3 changes: 3 additions & 0 deletions src/lib/onboard.ts
Original file line number Diff line number Diff line change
Expand Up @@ -1523,6 +1523,7 @@ const { getSandboxRuntimeRegistryFields, hasSandboxGpuDrift, updateReusedSandbox

async function createSandboxWithBaseImageResolution(
baseImageResolutionContext: import("./onboard/base-image-resolution-flow").BaseImageResolutionContext,
portableRuntimeAuthority: import("./state/onboard-checkpoint-types").CheckpointPortableRuntimeAuthority | null,
Comment thread
coderabbitai[bot] marked this conversation as resolved.
computePlan: import("./onboard/compute/plan").OpenShellComputePlan,
managedWorkloadRebuild: import("./onboard/workload/rebuild").ManagedWorkloadRebuildHandoff | null,
tempManagedRuntime: boolean,
Expand Down Expand Up @@ -1945,6 +1946,7 @@ async function createSandboxWithBaseImageResolution(
sandboxEnv,
sandboxStartupCommand,
lifecycleGeneration: createdSandboxLifecycle.generation,
portableRuntimeAuthority,
prebuild,
restoreBackupPath,
terminalAgent: agentDefs.isTerminalAgent(agent),
Expand Down Expand Up @@ -3556,6 +3558,7 @@ async function runOnboard(opts: OnboardOptions = {}): Promise<void> {
withSandboxPortReservationScope((dashboardPortReservationScope) =>
createSandboxWithBaseImageResolution(
baseImageResolutionContext,
lockedRuntime.preparedPortableAuthority,
onboardingComputePlan,
opts.managedWorkloadRebuild ?? null,
opts.tempManagedRuntime === true,
Expand Down
Loading
Loading