fix(portable): separate Podman activation readiness

docs/reference/commands.mdx

@@ -874,6 +874,22 @@ After all portable TCP probe attempts fail, onboarding prints commands for the u
 
 Onboarding prints the same commands when the portable probe cannot reach the user-scoped Podman service.
 The printed rerun command keeps the portable experimental profile selected.
+Portable commands reconstruct the current user's rootless Podman socket authority from NemoClaw state before they use the Docker-compatible API.
+They do not select an endpoint from ambient Docker or Podman runtime variables or named connections.
+When `podman.service` reports inactive and the recorded socket exists, NemoClaw first makes one 10-second API request through the guarded recorded authority.
+A valid server version classifies the endpoint as warm and avoids starting another socket service.
+A missing socket or a response without a valid server version enters bounded cold activation.
+Any socket authority change during this precheck fails at the socket authority stage.
+When the user-scoped socket-backed service needs activation, NemoClaw activates it and waits through a bounded startup period for a real Podman API response.
+During cold activation, the first API probe can cause systemd to replace the socket inode.
+NemoClaw requalifies one such replacement and repeats the probe only when the socket path, device, mode, owner, and complete directory authority remain unchanged.
+Any other authority change or a second inode replacement fails the readiness check.
+After cold activation succeeds, later API health checks use the fixed 10-second steady-state deadline.
+Onboarding and portable sandbox lifecycle commands use this same readiness contract.
+Failures identify socket authority, service activation, startup API health, or steady-state API health without reporting credentials.
+NemoClaw does not fall back to Docker or report an absent or unreachable endpoint as healthy.
+A successful cold path uses the `cold` timing label and reports activation, API, and total time in milliseconds.
+A successful warm path uses the `warm` timing label and reports steady-state API and total time in milliseconds.
 To tune the existing-gateway HTTP health poll, use `NEMOCLAW_REUSE_HEALTH_POLL_COUNT` (default `6`) and `NEMOCLAW_REUSE_HEALTH_POLL_INTERVAL` (default `5` seconds).
 The poll count is clamped to a minimum of `1` so the health probe always runs at least once, and the interval is clamped to a minimum of `0` (no sleep between attempts).
 
@@ -4968,6 +4984,7 @@ Defaults are sized for typical hardware; override only if you see false-positive
 | Variable | Default | Effect |
 |----------|---------|--------|
 | `NEMOCLAW_MCP_PROVIDER_SYNC_TIMEOUT_SECONDS` | `30` | Maximum time to wait for an OpenShell MCP provider credential revision to become active or fully revoked inside the sandbox. Integer seconds; raise only when provider synchronization is unusually slow. |
+| `NEMOCLAW_PORTABLE_PODMAN_STARTUP_TIMEOUT_MS` | `60000` | Maximum cold-start time for portable rootless Podman socket activation and the first real API response. Set an integer from `15000` through `300000` milliseconds. This setting does not change the fixed 10,000 ms steady-state API deadline. |
 | `NEMOCLAW_SANDBOX_EXEC_TIMEOUT_MS` | per call site (typically `15000`) | Overrides the default timeout for `openshell sandbox exec` calls issued by recovery and lifecycle helpers. Integer milliseconds; non-positive or non-numeric values fall back to the per-call-site default. |
 | `NEMOCLAW_STATUS_PROBE_TIMEOUT_MS` | built-in default | Overrides the timeout for the OpenShell status probe used by `$$nemoclaw <name> status`. Integer milliseconds; non-positive or non-numeric values fall back to the default. |
 | `NEMOCLAW_WSL_GPU_PROOF_TIMEOUT_MS` | `180000` | Maximum time for the bounded Docker CUDA workload on an eligible ARM64 Linux host. A positive finite number of milliseconds overrides the default. Invalid, infinite, zero, and negative values use the default. |

docs/reference/troubleshooting.mdx

@@ -3329,6 +3329,71 @@ Then rerun portable onboarding:
 $$nemoclaw onboard --experimental-profile portable
 ```
 
+### Portable Podman Readiness Fails
+
+Portable commands use the current user's rootless Podman socket authority recorded in NemoClaw state.
+They ignore ambient Docker and Podman runtime selectors, including named connections.
+Do not export another `DOCKER_HOST`, `DOCKER_CONTEXT`, `CONTAINER_HOST`, or `CONTAINER_CONNECTION` to bypass a readiness failure.
+
+When `podman.service` reports inactive and the recorded socket exists, NemoClaw first makes one 10-second API request through the guarded recorded authority.
+A valid server version classifies the endpoint as warm and avoids starting another socket service.
+A missing socket or a response without a valid server version enters bounded cold activation.
+Any socket authority change during this precheck fails at the socket authority stage and is not eligible for inode requalification.
+
+During cold activation, the first API probe can cause systemd to replace the socket inode.
+NemoClaw requalifies one such replacement and repeats the probe only when the socket path, device, mode, owner, and complete directory authority remain unchanged.
+Any other authority change or a second inode replacement fails at the socket authority stage.
+
+A portable readiness failure identifies the stage that did not complete:
+
+| Stage | Meaning | Recovery |
+| --- | --- | --- |
+| Socket authority | The portable lifecycle receipt is unsafe or invalid, its schema is `1` through `3` and predates recorded portable Podman authority, the recorded authority does not match the current Linux user, the recorded socket is unsafe, another authority field changed, or its inode was replaced more than once during cold activation. | If the error says that the receipt is unsafe or invalid, or that it predates recorded authority, run `$$nemoclaw onboard --experimental-profile portable`. If the recorded authority does not match the current Linux user, run NemoClaw as the user who created the portable state or rerun portable onboarding as the current user. Otherwise, stop and inspect the reported path. Restore the recorded current-user runtime instead of selecting another endpoint. |
+| Service activation | The current user's systemd manager could not activate the Podman socket-backed service, or activation did not create the recorded socket within the startup period. | Inspect `podman.socket` and `podman.service` with the commands below. Correct the reported user-unit failure, then rerun the NemoClaw command. |
+| Startup API health | The service was activated, but the recorded endpoint did not return a real Podman API response within the startup period. | Inspect the user-unit logs and run the explicit API request below against the reported socket path. Raise `NEMOCLAW_PORTABLE_PODMAN_STARTUP_TIMEOUT_MS` only when valid cold activation needs more than 60,000 ms. |
+| Steady-state API health | An endpoint that completed activation did not answer the later shorter health check. | Inspect host load and the user-unit logs, then rerun the NemoClaw command. |
+
+Inspect the current user's units without changing them:
+
+```bash
+systemctl --user status podman.socket podman.service --no-pager
+journalctl --user -u podman.socket -u podman.service --since -10m --no-pager
+```
+
+If the units need activation, restart the active service when present and start the socket for the current user session:
+
+```bash
+systemctl --user try-restart podman.service
+systemctl --user start podman.socket
+```
+
+These commands do not enable the socket for later user sessions.
+
+Use the exact socket path from the NemoClaw failure to require a real server response:
+
+```bash
+podman --remote \
+  --url unix://<recorded-socket-path> \
+  version \
+  --format 'Server Version: {{.Server.Version}}'
+```
+
+Continue only when the command exits with status `0` and prints a nonempty server version.
+The request and the readiness report contain no credentials.
+Rerun the original NemoClaw command without exporting a Docker or Podman runtime selector.
+
+If valid cold activation needs a larger budget, set an integer from `15000` through `300000` milliseconds:
+
+```bash
+export NEMOCLAW_PORTABLE_PODMAN_STARTUP_TIMEOUT_MS=120000
+$$nemoclaw <original-arguments>
+```
+
+The default cold-start budget is 60,000 ms.
+The later steady-state API deadline is fixed at 10,000 ms and does not use this setting.
+A successful cold path uses the `cold` timing label and reports activation, API, and total time in milliseconds.
+A successful warm path uses the `warm` timing label and reports steady-state API and total time in milliseconds.
+
 ### Portable Host Gateway Is Unreachable
 
 The portable experimental profile maps `host.openshell.internal` to the OpenShell Podman host gateway.
@@ -3344,15 +3409,15 @@ Portable onboarding reports output like this:
 ```
 
 If `podman.service` is active, restart it.
-Then enable and start the user-scoped Podman socket:
+Then start the user-scoped Podman socket for the current user session:
 
 ```bash
 systemctl --user try-restart podman.service
-systemctl --user enable --now podman.socket
+systemctl --user start podman.socket
 ```
 
 The first command does not start an inactive service.
-The second command enables and starts the current user's Podman API socket.
+The second command starts the current user's Podman API socket without enabling it for later user sessions.
 These commands affect only the current user's Podman units.
 They do not read or write credentials.
 
@@ -3368,6 +3433,9 @@ Expected output:
 active
 ```
 
+An active socket alone does not establish API health.
+Run the explicit Podman API request from [Portable Podman Readiness Fails](#portable-podman-readiness-fails) before you rerun onboarding.
+
 Then rerun portable onboarding:
 
 ```bash

src/lib/actions/sandbox/doctor-lifecycle-registration.test.ts

@@ -1,9 +1,45 @@
 // SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
 // SPDX-License-Identifier: Apache-2.0
 
-import { describe, expect, it } from "vitest";
+import { beforeEach, describe, expect, it, vi } from "vitest";
+
+const receiptReadinessMocks = vi.hoisted(() => ({
+  inspect: vi.fn(),
+}));
+
+vi.mock("../../onboard/experimental/portable-runtime-receipt-readiness", () => ({
+  inspectPortableRuntimeReceiptReadiness: receiptReadinessMocks.inspect,
+}));
+
+import type { PortablePodmanReadinessResult } from "../../onboard/experimental/portable-runtime-readiness";
 import type { SandboxEntry } from "../../state/registry";
-import { buildLifecycleRegistrationCheck } from "./doctor-lifecycle-registration";
+import {
+  buildLifecycleRegistrationCheck,
+  buildPortableRuntimeCheck,
+} from "./doctor-lifecycle-registration";
+
+const READY_PORTABLE_RUNTIME = {
+  ok: true,
+  authority: {
+    directoryChain: [],
+    device: "1",
+    inode: "2",
+    mode: String(0o140600),
+    ownerUid: "1001",
+    socketPath: "/run/user/1001/podman/podman.sock",
+  },
+  dockerHost: "unix:///run/user/1001/podman/podman.sock",
+  serverVersion: "5.6.1",
+  timing: { mode: "warm", activationMs: 0, apiMs: 7, totalMs: 7 },
+} satisfies PortablePodmanReadinessResult;
+
+const FAILED_PORTABLE_RUNTIME = {
+  ok: false,
+  stage: "startup API health",
+  detail: "Podman did not report a server version.",
+  socketPath: "/run/user/1001/podman/podman.sock",
+  timing: { mode: "cold", activationMs: 21, apiMs: 9, totalMs: 30 },
+} satisfies PortablePodmanReadinessResult;
 
 function sandbox(overrides: Partial<SandboxEntry> = {}): SandboxEntry {
   return {
@@ -24,6 +60,36 @@ function sandbox(overrides: Partial<SandboxEntry> = {}): SandboxEntry {
 }
 
 describe("doctor lifecycle registration checks", () => {
+  beforeEach(() => {
+    receiptReadinessMocks.inspect.mockReset();
+  });
+
+  it("renders server and timing detail for a ready portable Podman API", () => {
+    receiptReadinessMocks.inspect.mockReturnValue(READY_PORTABLE_RUNTIME);
+
+    expect(buildPortableRuntimeCheck("alpha")).toEqual({
+      group: "Host",
+      label: "Portable Podman API",
+      status: "ok",
+      detail: "server 5.6.1; warm; activation 0 ms; API 7 ms; total 7 ms",
+    });
+    expect(receiptReadinessMocks.inspect).toHaveBeenCalledWith("alpha");
+  });
+
+  it("renders the failure stage, recorded socket, and recovery hint", () => {
+    receiptReadinessMocks.inspect.mockReturnValue(FAILED_PORTABLE_RUNTIME);
+
+    expect(buildPortableRuntimeCheck("alpha")).toEqual({
+      group: "Host",
+      label: "Portable Podman API",
+      status: "fail",
+      detail:
+        "startup API health: Podman did not report a server version. Recorded socket: /run/user/1001/podman/podman.sock.",
+      hint: "repair the recorded current-user Podman endpoint, then retry",
+    });
+    expect(receiptReadinessMocks.inspect).toHaveBeenCalledWith("alpha");
+  });
+
   it("reports a complete managed sandbox registration as ok", () => {
     expect(buildLifecycleRegistrationCheck("alpha", sandbox(), "nemoclaw")).toMatchObject({
       group: "Sandbox",

src/lib/actions/sandbox/doctor-lifecycle-registration.ts

@@ -5,6 +5,7 @@ import {
   collectLifecycleRegistrationIssues,
   type LifecycleRegistrationIssue,
 } from "../../domain/lifecycle-registration";
+import { inspectPortableRuntimeReceiptReadiness } from "../../onboard/experimental/portable-runtime-receipt-readiness";
 import type { SandboxEntry } from "../../state/registry";
 import type { DoctorCheck } from "./doctor-report";
 
@@ -23,6 +24,28 @@ function formatFieldList(
     .join(", ");
 }
 
+export function buildPortableRuntimeCheck(sandboxName: string): DoctorCheck | null {
+  const portable = inspectPortableRuntimeReceiptReadiness(sandboxName);
+  if (!portable) return null;
+  const recordedSocket = !portable.ok && portable.socketPath
+    ? ` Recorded socket: ${portable.socketPath}.`
+    : "";
+  return portable.ok
+    ? {
+        group: "Host",
+        label: "Portable Podman API",
+        status: "ok",
+        detail: `server ${portable.serverVersion}; ${portable.timing.mode}; activation ${String(portable.timing.activationMs)} ms; API ${String(portable.timing.apiMs)} ms; total ${String(portable.timing.totalMs)} ms`,
+      }
+    : {
+        group: "Host",
+        label: "Portable Podman API",
+        status: "fail",
+        detail: `${portable.stage}: ${portable.detail}${recordedSocket}`,
+        hint: "repair the recorded current-user Podman endpoint, then retry",
+      };
+}
+
 export function buildLifecycleRegistrationCheck(
   sandboxName: string,
   entry: SandboxEntry,

src/lib/actions/sandbox/doctor.ts

@@ -49,7 +49,10 @@ import {
   type DoctorInferenceRoute,
   resolveDoctorReasoningEffort,
 } from "./doctor-inference";
-import { buildLifecycleRegistrationCheck } from "./doctor-lifecycle-registration";
+import {
+  buildLifecycleRegistrationCheck,
+  buildPortableRuntimeCheck,
+} from "./doctor-lifecycle-registration";
 import { collectMessagingDoctorChecks } from "./doctor-messaging";
 import {
   buildDoctorReport,
@@ -136,36 +139,44 @@ function cliBuildCheck(): DoctorCheck {
   };
 }
 
-function collectHostChecks(sb: SandboxEntry | null | undefined): {
-  checks: DoctorCheck[];
-  openshellBin: ReturnType<typeof resolveOpenshell>;
-} {
-  const cli = cliBuildCheck();
-  const openshellBin = resolveOpenshell();
-  let runtimeCheck: DoctorCheck;
+function inspectRuntimeHost(sb: SandboxEntry | null | undefined): DoctorCheck {
+  const portable = sb ? buildPortableRuntimeCheck(sb.name) : null;
+  if (portable) return portable;
+  const recorded = sb?.openshellDriver?.trim();
+  const provider = recorded
+    ? requireRuntimeProviderBundle(recorded, CURRENT_RUNTIME_PROVIDER_BUNDLES)
+    : resolveCurrentRuntimeProviderBundle();
+  return provider.preflightDoctor.inspectHost();
+}
+
+function runtimeHostCheck(sb: SandboxEntry | null | undefined): DoctorCheck {
   try {
-    const recorded = sb?.openshellDriver?.trim();
-    const provider = recorded
-      ? requireRuntimeProviderBundle(recorded, CURRENT_RUNTIME_PROVIDER_BUNDLES)
-      : resolveCurrentRuntimeProviderBundle();
-    runtimeCheck = provider.preflightDoctor.inspectHost();
+    return inspectRuntimeHost(sb);
   } catch (error) {
     const detail =
       error instanceof RuntimeProviderSelectionError
         ? error.message
         : `Runtime provider inspection failed: ${error instanceof Error ? error.message : String(error)}`;
-    runtimeCheck = {
+    return {
       group: "Host",
       label: "Runtime provider",
       status: "fail",
       detail,
       hint: "restore a supported durable runtime provider identity before retrying",
     };
   }
+}
+
+function collectHostChecks(sb: SandboxEntry | null | undefined): {
+  checks: DoctorCheck[];
+  openshellBin: ReturnType<typeof resolveOpenshell>;
+} {
+  const cli = cliBuildCheck();
+  const openshellBin = resolveOpenshell();
   return {
     checks: [
       cli,
-      runtimeCheck,
+      runtimeHostCheck(sb),
       {
         group: "Host",
         label: "OpenShell CLI",

src/lib/actions/sandbox/gateway-failure-classifier.ts

@@ -9,11 +9,20 @@ import { CLI_NAME } from "../../cli/branding";
 import { GATEWAY_PORT } from "../../core/ports";
 import { resolveSandboxContainerOwner } from "../../domain/sandbox/container-owner";
 import { resolveGatewayPortFromName } from "../../onboard/gateway-binding";
+import type { PortablePodmanReadinessResult } from "../../onboard/experimental/portable-runtime-readiness";
+import {
+  inspectPortableRuntimeReceiptReadiness,
+  type PortableRuntimeReceiptReadinessDeps,
+} from "../../onboard/experimental/portable-runtime-receipt-readiness";
 import * as registry from "../../state/registry";
 import { getSandboxTargetGatewayName } from "./gateway-target";
 
 const DOCKER_TIMEOUT_MS = 3000;
 const PORT_PROBE_TIMEOUT_MS = 2000;
+const portableRuntimeFailures = new Map<
+  string,
+  Extract<PortablePodmanReadinessResult, { ok: false }>
+>();
 
 export type GatewayFailureLayer =
   | "docker_unreachable"
@@ -281,8 +290,22 @@ export function isDockerRuntimeDown(
   opts?: {
     runners?: Pick<GatewayFailureRunners, "dockerInfo">;
     getSandbox?: SandboxDriverLookup;
+    portableLifecycle?: PortableRuntimeReceiptReadinessDeps;
   },
 ): boolean {
+  const portable = inspectPortableRuntimeReceiptReadiness(sandboxName, opts?.portableLifecycle);
+  if (portable) {
+    if (portable.ok) {
+      portableRuntimeFailures.delete(sandboxName);
+      console.log(
+        `  Portable Podman readiness: ${portable.timing.mode}; activation ${String(portable.timing.activationMs)} ms; API ${String(portable.timing.apiMs)} ms; total ${String(portable.timing.totalMs)} ms.`,
+      );
+      return false;
+    }
+    portableRuntimeFailures.set(sandboxName, portable);
+    return true;
+  }
+  portableRuntimeFailures.delete(sandboxName);
   const getSandbox = opts?.getSandbox ?? registry.getSandbox;
   if (!isDockerBackedSandbox(sandboxName, getSandbox)) return false;
   const probe = opts?.runners?.dockerInfo ?? defaultRunners.dockerInfo;
@@ -302,6 +325,25 @@ export function printDockerRuntimeDownGuidance(
 ): void {
   const writer = opts.writer ?? console.error;
   const retryCommand = opts.retryCommand ?? "status";
+  const portable = portableRuntimeFailures.get(sandboxName);
+  portableRuntimeFailures.delete(sandboxName);
+  if (portable) {
+    writer(`  Failure stage: ${portable.stage} — ${portable.detail}`);
+    if (portable.socketPath) writer(`  Recorded socket: ${portable.socketPath}`);
+    writer(
+      `  Portable Podman readiness (${portable.timing.mode}): activation ${String(portable.timing.activationMs)} ms; API ${String(portable.timing.apiMs)} ms; total ${String(portable.timing.totalMs)} ms.`,
+    );
+    writer(
+      `  The receipt-owned Podman endpoint for sandbox '${sandboxName}' is not ready; no Docker or named-connection fallback was used.`,
+    );
+    writer("  Recovery:");
+    writer(
+      "    1. Check the reported readiness stage and the current user's Podman socket service.",
+    );
+    writer("    2. Confirm the recorded endpoint returns a real Podman server version.");
+    writer(`    3. Retry: ${CLI_NAME} ${sandboxName} ${retryCommand}`);
+    return;
+  }
   writer(`  ${getLayerHeader("docker_unreachable")}`);
   writer(
     `  The Docker daemon is not reachable, so sandbox '${sandboxName}' cannot be verified or started.`,