fix(portable): route receipt failures to valid recovery (#9194)

<!-- markdownlint-disable MD041 -->
## Summary

Portable readiness failures without a recorded socket path no longer
send users to repair a nonexistent endpoint. Invalid and legacy receipts
now route to portable onboarding, current-user authority mismatches
route to the recorded user or current-user onboarding, and socket
service diagnostics appear only when a validated socket path is
available.

## Related Issue

Follow-up to #9186 for #9070.

## Changes

- Record the required no-socket recovery in the portable readiness
result so doctor and gateway consumers do not infer endpoint repair from
error text.
- Route invalid and legacy receipts to `nemoclaw onboard
--experimental-profile portable`.
- Route a current-user authority mismatch to the recorded user or
onboarding as the current user.
- Keep socket service and API diagnostics conditional on a reported
socket path.
- Add doctor and gateway regression tests for invalid and legacy receipt
recovery.
- Update troubleshooting guidance to match the runtime recovery
contract.

## Type of Change

- [ ] Code change (feature, bug fix, or refactor)
- [x] Code change with doc updates
- [ ] Doc only (prose changes, no code sample modifications)
- [ ] Doc only (includes code sample changes)

## Quality Gates

- [x] Tests added or updated for changed behavior
- [ ] Existing tests cover changed behavior — justification:
- [ ] Tests not applicable — justification:
- [x] Docs updated for user-facing behavior changes
- [ ] Docs not applicable — justification:
- [x] Sensitive paths changed (security, policy, credentials, preflight,
onboarding, inference, runner, sandbox, or messaging)
- [x] Sensitive-path review completed or maintainer-approved waiver
recorded — reviewer/approval link/justification: Independent Codex
Desktop review passed all nine security categories for commit under
review `5c6a000f5e6d036845cfbebee2a7fd4d0000c36d`. Invalid, legacy, and
current-user mismatch paths fail closed without fabricated endpoint
repair. The doctor and gateway tests cover each recovery branch.
- [ ] Non-success, skipped, or missing CI check accepted by maintainer —
check name, approval link, and follow-up issue:

## Documentation Writer Review

- [x] Documentation writer subagent reviewed the completed changes
- Result: `docs-updated`
- Evidence: `docs/reference/troubleshooting.mdx` routes unsafe, invalid,
and legacy receipts to portable onboarding; routes current-user
mismatches to the recorded user or current-user onboarding; and limits
socket service and API inspection to failures that report a socket path.
Doctor and gateway guidance match these paths. The merge preserves the
reviewed patch and adds current-base Google Gemini documentation and
provider-model tests outside the nine PR-owned files. Focused recovery
tests passed 59/59. `npm run docs` completed with 0 errors and 2
existing warnings. `git diff --check` passed.
- Agent: `Codex Desktop`
<!-- docs-review-head-sha: 5c6a000 -->
<!-- docs-review-agents-blob-sha: e30afb2 -->

## DGX Station Hardware Evidence

- [ ] Tested on DGX Station
- Tested commit:
- Station profile/scenario:
- Result:
- Supporting evidence:

## Verification

- [x] PR description includes a `Signed-off-by:` line and every commit
appears as `Verified` in GitHub
- [x] Normal `pre-commit`, `commit-msg`, and `pre-push` hooks passed, or
`npm run validate:pr` passed after refreshing `origin/main` when hooks
were skipped or unavailable
- [x] Targeted behavior tests pass for the current change set, or tests
are marked not applicable above — `npx vitest run --project cli
--project integration
src/lib/actions/sandbox/doctor-lifecycle-registration.test.ts
src/lib/onboard/experimental/portable-runtime-readiness.test.ts
test/gateway-failure-classifier.test.ts`: 59 passed
- [ ] Applicable broad gate passed — `npm test` for broad
runtime/test-harness changes; `npm run check` for repo-wide
validation/coverage changes — command/result:
- [x] Quality Gates section completed with required justifications or
waivers
- [x] No secrets, API keys, or credentials committed
- [ ] `npm run docs` builds without warnings (doc changes only)
- [x] Doc pages follow the [style
guide](https://github.com/NVIDIA/NemoClaw/blob/main/docs/CONTRIBUTING.md)
(doc changes only)
- [ ] New doc pages include SPDX header and frontmatter (new pages only)

---
Signed-off-by: Prekshi Vyas <prekshiv@nvidia.com>


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Bug Fixes**
* Improved Portable Podman troubleshooting for missing sockets, legacy
receipts, and user-authority mismatches.
* Recovery guidance now directs users to portable onboarding or the
appropriate recorded/current user path.
* Added clearer instructions for inspecting Podman units and validating
the reported socket.
* Prevented inappropriate endpoint-repair guidance and credential
exposure in affected scenarios.

* **Tests**
* Expanded coverage for portable onboarding failures, legacy receipts,
socket handling, and authority mismatches.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

Signed-off-by: Prekshi Vyas <prekshiv@nvidia.com>
Co-authored-by: Carlos Villela <cvillela@nvidia.com>

Author: prekshivyas

docs/reference/troubleshooting.mdx

@@ -3353,7 +3353,10 @@ A portable readiness failure identifies the stage that did not complete:
 | Startup API health | The service was activated, but the recorded endpoint did not return a real Podman API response within the startup period. | Inspect the user-unit logs and run the explicit API request below against the reported socket path. Raise `NEMOCLAW_PORTABLE_PODMAN_STARTUP_TIMEOUT_MS` only when valid cold activation needs more than 60,000 ms. |
 | Steady-state API health | An endpoint that completed activation did not answer the later shorter health check. | Inspect host load and the user-unit logs, then rerun the NemoClaw command. |
 
-Inspect the current user's units without changing them:
+The remaining service and API inspection applies only when the failure reports a recorded socket path.
+If the failure does not report one, follow the recovery in the table and do not try another endpoint.
+
+When a recorded socket path is reported, inspect the current user's units without changing them:
 
 ```bash
 systemctl --user status podman.socket podman.service --no-pager
@@ -3369,7 +3372,7 @@ systemctl --user start podman.socket
 
 These commands do not enable the socket for later user sessions.
 
-Use the exact socket path from the NemoClaw failure to require a real server response:
+Then use the exact socket path from the NemoClaw failure to require a real server response:
 
 ```bash
 podman --remote \

src/lib/actions/sandbox/doctor-lifecycle-registration.test.ts

@@ -41,6 +41,30 @@ const FAILED_PORTABLE_RUNTIME = {
   timing: { mode: "cold", activationMs: 21, apiMs: 9, totalMs: 30 },
 } satisfies PortablePodmanReadinessResult;
 
+const PORTABLE_ONBOARDING_FAILURES = [
+  {
+    name: "invalid",
+    result: {
+      ok: false,
+      stage: "socket authority",
+      detail: "The portable lifecycle receipt is unsafe or invalid; rerun onboarding.",
+      recovery: "portable-onboarding",
+      timing: { mode: "warm", activationMs: 0, apiMs: 0, totalMs: 0 },
+    } satisfies PortablePodmanReadinessResult,
+  },
+  {
+    name: "legacy",
+    result: {
+      ok: false,
+      stage: "socket authority",
+      detail:
+        "The lifecycle receipt predates recorded portable Podman authority; rerun onboarding.",
+      recovery: "portable-onboarding",
+      timing: { mode: "warm", activationMs: 0, apiMs: 0, totalMs: 0 },
+    } satisfies PortablePodmanReadinessResult,
+  },
+] as const;
+
 function sandbox(overrides: Partial<SandboxEntry> = {}): SandboxEntry {
   return {
     name: "alpha",
@@ -90,6 +114,36 @@ describe("doctor lifecycle registration checks", () => {
     expect(receiptReadinessMocks.inspect).toHaveBeenCalledWith("alpha");
   });
 
+  it.each(PORTABLE_ONBOARDING_FAILURES)(
+    "sends a $name receipt failure to portable onboarding without endpoint repair",
+    ({ result }) => {
+      receiptReadinessMocks.inspect.mockReturnValue(result);
+
+      const check = buildPortableRuntimeCheck("alpha");
+
+      expect(check).toMatchObject({
+        status: "fail",
+        detail: expect.not.stringContaining("Recorded socket"),
+        hint: "rerun portable onboarding with `nemoclaw onboard --experimental-profile portable`, then retry",
+      });
+      expect(check?.hint).not.toContain("endpoint");
+    },
+  );
+
+  it("routes a current-user authority mismatch to its recorded user or current-user onboarding", () => {
+    receiptReadinessMocks.inspect.mockReturnValue({
+      ok: false,
+      stage: "socket authority",
+      detail: "The recorded portable Podman authority does not match the current Linux user.",
+      recovery: "current-user-authority",
+      timing: { mode: "warm", activationMs: 0, apiMs: 0, totalMs: 0 },
+    } satisfies PortablePodmanReadinessResult);
+
+    expect(buildPortableRuntimeCheck("alpha")).toMatchObject({
+      hint: "run NemoClaw as the user who created the portable state, or rerun portable onboarding as the current user",
+    });
+  });
+
   it("reports a complete managed sandbox registration as ok", () => {
     expect(buildLifecycleRegistrationCheck("alpha", sandbox(), "nemoclaw")).toMatchObject({
       group: "Sandbox",

src/lib/actions/sandbox/doctor-lifecycle-registration.ts

@@ -27,9 +27,14 @@ function formatFieldList(
 export function buildPortableRuntimeCheck(sandboxName: string): DoctorCheck | null {
   const portable = inspectPortableRuntimeReceiptReadiness(sandboxName);
   if (!portable) return null;
-  const recordedSocket = !portable.ok && portable.socketPath
-    ? ` Recorded socket: ${portable.socketPath}.`
-    : "";
+  const recordedSocket =
+    !portable.ok && portable.socketPath ? ` Recorded socket: ${portable.socketPath}.` : "";
+  const recoveryHint =
+    !portable.ok && !portable.socketPath
+      ? portable.recovery === "current-user-authority"
+        ? "run NemoClaw as the user who created the portable state, or rerun portable onboarding as the current user"
+        : "rerun portable onboarding with `nemoclaw onboard --experimental-profile portable`, then retry"
+      : "repair the recorded current-user Podman endpoint, then retry";
   return portable.ok
     ? {
         group: "Host",
@@ -42,7 +47,7 @@ export function buildPortableRuntimeCheck(sandboxName: string): DoctorCheck | nu
         label: "Portable Podman API",
         status: "fail",
         detail: `${portable.stage}: ${portable.detail}${recordedSocket}`,
-        hint: "repair the recorded current-user Podman endpoint, then retry",
+        hint: recoveryHint,
       };
 }
 

src/lib/actions/sandbox/gateway-failure-classifier.ts

@@ -337,6 +337,19 @@ export function printDockerRuntimeDownGuidance(
       `  The receipt-owned Podman endpoint for sandbox '${sandboxName}' is not ready; no Docker or named-connection fallback was used.`,
     );
     writer("  Recovery:");
+    if (!portable.socketPath) {
+      if (portable.recovery === "current-user-authority") {
+        writer(
+          "    1. Run NemoClaw as the user who created the portable state, or rerun portable onboarding as the current user.",
+        );
+      } else {
+        writer(
+          `    1. Rerun portable onboarding: ${CLI_NAME} onboard --experimental-profile portable`,
+        );
+      }
+      writer(`    2. Retry: ${CLI_NAME} ${sandboxName} ${retryCommand}`);
+      return;
+    }
     writer(
       "    1. Check the reported readiness stage and the current user's Podman socket service.",
     );

src/lib/onboard/experimental/portable-demo-lifecycle.ts

@@ -528,6 +528,7 @@ function inspectReceiptRuntimeReadiness(
       stage: "socket authority",
       detail:
         "The lifecycle receipt predates recorded portable Podman authority; rerun onboarding.",
+      recovery: "portable-onboarding",
       timing: { mode: "warm", activationMs: 0, apiMs: 0, totalMs: 0 },
     };
   }

src/lib/onboard/experimental/portable-runtime-readiness.test.ts

@@ -309,7 +309,11 @@ describe("portable Podman activation readiness", () => {
     expect(validFailure.ok ? "" : portablePodmanReadinessError(validFailure).message).toContain(
       `Recorded socket: ${AUTHORITY.socketPath}.`,
     );
-    expect(invalidFailure).toMatchObject({ ok: false, stage: "socket authority" });
+    expect(invalidFailure).toMatchObject({
+      ok: false,
+      stage: "socket authority",
+      recovery: "current-user-authority",
+    });
     expect(invalidFailure).not.toHaveProperty("socketPath");
   });
 

src/lib/onboard/experimental/portable-runtime-readiness.ts

@@ -66,6 +66,8 @@ export interface PortablePodmanReadinessTiming {
   readonly totalMs: number;
 }
 
+export type PortablePodmanReadinessRecovery = "portable-onboarding" | "current-user-authority";
+
 export type PortablePodmanReadinessResult =
   | {
       readonly ok: true;
@@ -79,6 +81,7 @@ export type PortablePodmanReadinessResult =
       readonly stage: PortablePodmanReadinessStage;
       readonly detail: string;
       readonly socketPath?: string;
+      readonly recovery?: PortablePodmanReadinessRecovery;
       readonly timing: PortablePodmanReadinessTiming;
     };
 
@@ -171,12 +174,14 @@ function failure(
   apiMs: number,
   totalMs: number,
   socketPath?: string,
+  recovery?: PortablePodmanReadinessRecovery,
 ): PortablePodmanReadinessResult {
   return {
     ok: false,
     stage,
     detail,
     ...(socketPath ? { socketPath } : {}),
+    ...(recovery ? { recovery } : {}),
     timing: timing(mode, activationMs, apiMs, totalMs),
   };
 }
@@ -291,6 +296,8 @@ export function inspectPortablePodmanReadiness(
       0,
       0,
       elapsed(now, startedAt),
+      undefined,
+      "current-user-authority",
     );
   }
 

src/lib/onboard/experimental/portable-runtime-receipt-readiness.ts

@@ -175,6 +175,7 @@ export function inspectPortableRuntimeReceiptReadiness(
       ok: false,
       stage: "socket authority",
       detail: "The portable lifecycle receipt is unsafe or invalid; rerun onboarding.",
+      recovery: "portable-onboarding",
       timing: { mode: "warm", activationMs: 0, apiMs: 0, totalMs: 0 },
     };
   }
@@ -185,6 +186,7 @@ export function inspectPortableRuntimeReceiptReadiness(
       stage: "socket authority",
       detail:
         "The lifecycle receipt predates recorded portable Podman authority; rerun onboarding.",
+      recovery: "portable-onboarding",
       timing: { mode: "warm", activationMs: 0, apiMs: 0, totalMs: 0 },
     };
   }

test/gateway-failure-classifier.test.ts

@@ -56,6 +56,23 @@ function writePortableReceipt(stateDir: string): void {
   );
 }
 
+function writeLegacyPortableReceipt(stateDir: string): void {
+  const receiptPath = portableDemoLifecycleInternals.receiptPath("alpha", stateDir);
+  fs.mkdirSync(path.dirname(receiptPath), { recursive: true });
+  fs.writeFileSync(
+    receiptPath,
+    `${JSON.stringify({
+      schemaVersion: 3,
+      sandboxName: "alpha",
+      sandboxId: "sandbox-id-alpha",
+      containerId: "a".repeat(64),
+      dashboardPort: 18789,
+      registryGeneration: "a".repeat(64),
+    })}\n`,
+    { mode: 0o600 },
+  );
+}
+
 function makeRunners(overrides: Partial<GatewayFailureRunners> = {}): GatewayFailureRunners {
   return {
     dockerInfo: () => true,
@@ -215,9 +232,7 @@ describe("isDockerRuntimeDown", () => {
       const out: string[] = [];
       printDockerRuntimeDownGuidance("alpha", { writer: (line) => out.push(line) });
       expect(out.join("\n")).toContain("steady-state API health");
-      expect(out.join("\n")).toContain(
-        `Recorded socket: ${PORTABLE_SOCKET_AUTHORITY.socketPath}`,
-      );
+      expect(out.join("\n")).toContain(`Recorded socket: ${PORTABLE_SOCKET_AUTHORITY.socketPath}`);
       expect(out.join("\n")).toContain("no Docker or named-connection fallback was used");
     } finally {
       fs.rmSync(stateDir, { recursive: true, force: true });
@@ -241,8 +256,38 @@ describe("isDockerRuntimeDown", () => {
       expect(dockerInfo).not.toHaveBeenCalled();
       const out: string[] = [];
       printDockerRuntimeDownGuidance("alpha", { writer: (line) => out.push(line) });
-      expect(out.join("\n")).toContain("socket authority");
-      expect(out.join("\n")).not.toContain("super-secret");
+      const guidance = out.join("\n");
+      expect(guidance).toContain("socket authority");
+      expect(guidance).toContain("nemoclaw onboard --experimental-profile portable");
+      expect(guidance).not.toContain("current user's Podman socket service");
+      expect(guidance).not.toContain("Confirm the recorded endpoint");
+      expect(guidance).not.toContain("super-secret");
+    } finally {
+      fs.rmSync(stateDir, { recursive: true, force: true });
+    }
+  });
+
+  it("routes a legacy portable receipt to onboarding without endpoint repair (#9070)", () => {
+    const stateDir = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-runtime-readiness-"));
+    writeLegacyPortableReceipt(stateDir);
+    const dockerInfo = vi.fn(() => true);
+    try {
+      expect(
+        isDockerRuntimeDown("alpha", {
+          runners: { dockerInfo },
+          getSandbox: dockerSandbox,
+          portableLifecycle: { stateDir },
+        }),
+      ).toBe(true);
+      expect(dockerInfo).not.toHaveBeenCalled();
+
+      const out: string[] = [];
+      printDockerRuntimeDownGuidance("alpha", { writer: (line) => out.push(line) });
+      const guidance = out.join("\n");
+      expect(guidance).toContain("predates recorded portable Podman authority");
+      expect(guidance).toContain("nemoclaw onboard --experimental-profile portable");
+      expect(guidance).not.toContain("current user's Podman socket service");
+      expect(guidance).not.toContain("Confirm the recorded endpoint");
     } finally {
       fs.rmSync(stateDir, { recursive: true, force: true });
     }