Merge remote-tracking branch 'origin/main' into feat/n1x-express-install-8574

Author: sandl99

docs/manage-sandboxes/uninstall-nemoclaw.mdx

@@ -115,6 +115,13 @@ Rerun `NEMOCLAW_GATEWAY_PORT=<port> $$nemoclaw uninstall` with the gateway port
 For an externally supervised authority, uninstall preserves the local gateway state used by the running process in both full and gateway-scoped cleanup.
 It also preserves the gateway process, supervisor resources, marked Linux unit, Docker resources, OpenShell binaries, and the declared external state directory.
 A custom-port uninstall does not stop or remove the default gateway service or its environment file.
+Uninstall does not stop an `openshell-gateway` process that another non-root user owns and that this installation did not record.
+It names the owner and process ID, leaves that process running, and continues with the remaining cleanup.
+If no other cleanup fails, uninstall exits with status `0` even though that process can keep its port in use.
+Uninstall still tries to stop a `root`-owned process and the gateway process that this installation recorded.
+If either of those stops fails, uninstall prints `sudo kill -9 <pid>` for the process.
+A gateway-scoped uninstall and every `--all-gateway-ports` pass exit nonzero after that failure.
+A single full uninstall reports the process and continues.
 Before scoped cleanup stops a Docker gateway process, including a managed default gateway service, NemoClaw requires two Docker namespace proofs.
 The selected Docker gateway configuration and any running gateway process must use the state-root-specific OpenShell sandbox namespace that NemoClaw generated.
 Because the supported OpenShell Podman schema does not expose `sandbox_namespace`, scoped Podman uninstall fails closed before signaling and preserves the gateway runtime evidence and local state.

docs/reference/commands.mdx

@@ -4082,6 +4082,8 @@ For a managed dual-Station vLLM runtime, full uninstall revalidates the exact re
 If that cleanup fails, uninstall exits nonzero, preserves its owner-only cleanup receipt, and tells you to resolve the reported peer error before retrying.
 Pair cleanup can partially complete before an error; verify both Stations before the retry.
 For an authenticated host-local vLLM runtime, full uninstall verifies the exact named container, NemoClaw ownership label, persisted API key, and authentication fingerprint before removing the container by its inspected ID.
+When that ownership state is missing, full uninstall removes the reserved `nemoclaw-vllm` container only when Docker reports its NemoClaw managed label and a valid container ID.
+An unlabeled container or malformed inspection remains in place and stops the remaining uninstall steps.
 For managed llama.cpp, full uninstall verifies the exact named container and network ownership before removing both resources by their inspected IDs.
 These host-local checks run before NemoClaw deletes their state.
 If Docker is unavailable or a resource does not match its persisted ownership state, uninstall exits nonzero before the remaining uninstall steps and preserves that state for recovery.
@@ -4151,6 +4153,13 @@ Rerun `NEMOCLAW_GATEWAY_PORT=<port> $$nemoclaw uninstall` with the gateway port
 For an externally supervised authority, uninstall preserves the selected local gateway state in both full and gateway-scoped cleanup.
 It also preserves the gateway process, supervisor resources, marked Linux unit, Docker resources, OpenShell binaries, and the declared external state directory.
 A custom-port uninstall does not stop or remove the default gateway service or its environment file.
+Uninstall does not stop an `openshell-gateway` process that another non-root user owns and that this installation did not record.
+It names the owner and process ID, leaves that process running, and continues with the remaining cleanup.
+If no other cleanup fails, uninstall exits with status `0` even though that process can keep its port in use.
+Uninstall still tries to stop a `root`-owned process and the gateway process that this installation recorded.
+If either of those stops fails, uninstall prints `sudo kill -9 <pid>` for the process.
+A gateway-scoped uninstall and every `--all-gateway-ports` pass exit nonzero after that failure.
+A single full uninstall reports the process and continues.
 Before scoped cleanup stops a Docker gateway process, including a managed default gateway service, NemoClaw requires two Docker namespace proofs.
 The selected Docker gateway configuration and any running gateway process must use the state-root-specific OpenShell sandbox namespace that NemoClaw generated.
 Because the supported OpenShell Podman schema does not expose `sandbox_namespace`, scoped Podman uninstall fails closed before signaling and preserves the gateway runtime evidence and local state.

docs/reference/troubleshooting.mdx

@@ -432,6 +432,9 @@ openshell gateway list
 
 If the gateway name and its port-scoped state remain, treat it as a second environment and select that port for cleanup.
 If the gateway is absent but the port still listens, cleanup did not stop the listener; follow the process or service remediation printed by uninstall before you retry.
+If uninstall reported that it kept an `openshell-gateway` process owned by another user running, that process still holds the port.
+This can happen after uninstall exits successfully because NemoClaw does not treat another user's process as a cleanup failure.
+Ask that user to stop the process, or onboard under a different `NEMOCLAW_GATEWAY_PORT`.
 
 Remove one environment by selecting its port:
 

src/lib/actions/uninstall/run-plan-foreign-user-gateway.test.ts

@@ -0,0 +1,99 @@
+// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+// SPDX-License-Identifier: Apache-2.0
+
+import fs from "node:fs";
+import os from "node:os";
+import path from "node:path";
+
+import { describe, expect, it, vi } from "vitest";
+
+import { type RunResult, runUninstallPlan } from "./run-plan";
+
+const HOST_GATEWAY_PID = 9999043;
+
+function ok(stdout = ""): RunResult {
+  return { status: 0, stdout, stderr: "" };
+}
+
+function notFound(): RunResult {
+  return { status: 1, stdout: "", stderr: "" };
+}
+
+function uninstallWithHostGatewayOwnedBy(uid: number): {
+  errors: string[];
+  exitCode: number;
+} {
+  const tmpHome = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-uninstall-foreign-"));
+  const errors: string[] = [];
+  const psResults = new Map<string, RunResult>([
+    ["stat=", ok("S\n")],
+    ["args=", ok("/usr/local/bin/openshell-gateway\n")],
+    ["user=", ok("otheruser\n")],
+    ["uid=", ok(`${uid}\n`)],
+  ]);
+  const run = (command: string, args: string[]): RunResult =>
+    command === "pgrep"
+      ? args.some((arg) => arg.includes("openshell-gateway"))
+        ? ok(`${HOST_GATEWAY_PID}\n`)
+        : notFound()
+      : command === "ps"
+        ? (psResults.get(args.at(-1) ?? "") ?? notFound())
+        : command === "openshell" && args.join(" ") === "gateway list -o json"
+          ? ok("[]")
+          : ok();
+  try {
+    const result = runUninstallPlan(
+      { assumeYes: true, deleteModels: false, keepOpenShell: false },
+      {
+        commandExists: (command) => command === "pgrep" || command === "openshell",
+        env: { HOME: tmpHome, NO_COLOR: "1" },
+        error: (message) => errors.push(message),
+        existsSync: () => false,
+        isTty: false,
+        kill: () => false,
+        log: vi.fn(),
+        requireCompleteGatewayProcessCleanup: true,
+        resolveGatewayTeardownAuthority: ({ gatewayName, gatewayPort }) => ({
+          gatewayName,
+          gatewayPort,
+          mode: "nemoclaw-managed",
+          source: "standalone",
+          endpoint: null,
+          stateDir: null,
+          supervisor: null,
+          requiredCapabilities: [],
+        }),
+        rmSync: vi.fn(),
+        run,
+        runDocker: () => ok(),
+      },
+    );
+    return { errors, exitCode: result.exitCode };
+  } finally {
+    fs.rmSync(tmpHome, { force: true, recursive: true });
+  }
+}
+
+describe("uninstall with a host gateway owned by another user", () => {
+  it("completes when the only unstoppable gateway process belongs to another user", () => {
+    const { errors, exitCode } = uninstallWithHostGatewayOwnedBy((process.getuid?.() ?? 0) + 1);
+
+    expect(exitCode).toBe(0);
+    expect(errors).toContainEqual(
+      `Kept otheruser-owned host openshell-gateway process ${HOST_GATEWAY_PID} running. ` +
+        "Cleanup does not stop a gateway process that another user owns.",
+    );
+    expect(errors).not.toContainEqual(
+      "Cannot continue uninstall because host gateway process cleanup did not complete.",
+    );
+  });
+
+  it("still fails when the current user's own gateway process cannot be stopped", () => {
+    const { errors, exitCode } = uninstallWithHostGatewayOwnedBy(process.getuid?.() ?? 0);
+
+    expect(exitCode).toBe(1);
+    expect(errors).toContainEqual(
+      "Cannot continue uninstall because host gateway process cleanup did not complete.",
+    );
+  });
+});

src/lib/actions/uninstall/run-plan-local-model-profile.test.ts

@@ -26,6 +26,20 @@ function notFound(): RunResult {
   return { status: 1, stdout: "", stderr: "" };
 }
 
+function dockerResults(results: ReadonlyMap<string, RunResult>) {
+  return vi.fn((args: string[]) => results.get(JSON.stringify(args)) ?? ok());
+}
+
+const ORPHANED_VLLM_INSPECT_ARGS = [
+  "container",
+  "inspect",
+  "--format",
+  '{{.Id}} {{index .Config.Labels "com.nvidia.nemoclaw.managed-vllm"}}',
+  "nemoclaw-vllm",
+];
+
+const RESERVED_INFERENCE_NAMES_ARGS = ["ps", "-a", "--format", "{{.Names}}"];
+
 function runUninstallPlan(options: UninstallRunOptions, deps: UninstallRunDeps) {
   return runUninstallPlanBase(options, {
     resolveGatewayTeardownAuthority: ({ gatewayName, gatewayPort }) => ({
@@ -79,9 +93,134 @@ function publishManagedLlamaOwner(
 }
 
 describe("uninstall local model profile cleanup", () => {
+  it("removes an orphaned host-local vLLM container only when its managed label is present (#8981)", () => {
+    const containerId = "a".repeat(64);
+    const runDocker = dockerResults(
+      new Map([[JSON.stringify(ORPHANED_VLLM_INSPECT_ARGS), ok(`${containerId} true\n`)]]),
+    );
+
+    const result = runUninstallPlan(
+      { assumeYes: true, deleteModels: false, keepOpenShell: true },
+      {
+        commandExists: (command) => command === "openshell" || command === "docker",
+        env: { HOME: "/tmp/nemoclaw-uninstall-orphaned-vllm" } as NodeJS.ProcessEnv,
+        existsSync: () => false,
+        isTty: false,
+        log: () => {},
+        run: vi.fn(okWithKnownGatewayList),
+        runDocker,
+      },
+    );
+
+    expect(result.exitCode).toBe(0);
+    expect(runDocker).toHaveBeenCalledWith(
+      ["rm", "-f", containerId],
+      expect.objectContaining({ timeout: 10_000 }),
+    );
+  });
+
+  it("preserves an orphaned host-local vLLM container without the managed label (#8981)", () => {
+    const errors: string[] = [];
+    const runDocker = dockerResults(
+      new Map([
+        [JSON.stringify(ORPHANED_VLLM_INSPECT_ARGS), ok(`${"b".repeat(64)} false\n`)],
+        [JSON.stringify(RESERVED_INFERENCE_NAMES_ARGS), ok("nemoclaw-vllm\n")],
+      ]),
+    );
+
+    const result = runUninstallPlan(
+      { assumeYes: true, deleteModels: false, keepOpenShell: true },
+      {
+        commandExists: (command) => command === "openshell" || command === "docker",
+        env: { HOME: "/tmp/nemoclaw-uninstall-unlabeled-vllm" } as NodeJS.ProcessEnv,
+        existsSync: () => false,
+        error: (message) => errors.push(message),
+        isTty: false,
+        log: () => {},
+        run: vi.fn(okWithKnownGatewayList),
+        runDocker,
+      },
+    );
+
+    expect(result.exitCode).toBe(1);
+    expect(runDocker.mock.calls.some(([args]) => args[0] === "rm")).toBe(false);
+    expect(errors.join("\n")).toContain("remains after ownership-aware cleanup");
+  });
+
+  it("preserves an orphaned host-local vLLM container after malformed inspection output (#8981)", () => {
+    const errors: string[] = [];
+    const runDocker = dockerResults(
+      new Map([
+        [JSON.stringify(ORPHANED_VLLM_INSPECT_ARGS), ok("not-a-container-id true\n")],
+        [JSON.stringify(RESERVED_INFERENCE_NAMES_ARGS), ok("nemoclaw-vllm\n")],
+      ]),
+    );
+
+    const result = runUninstallPlan(
+      { assumeYes: true, deleteModels: false, keepOpenShell: true },
+      {
+        commandExists: (command) => command === "openshell" || command === "docker",
+        env: { HOME: "/tmp/nemoclaw-uninstall-malformed-vllm" } as NodeJS.ProcessEnv,
+        existsSync: () => false,
+        error: (message) => errors.push(message),
+        isTty: false,
+        log: () => {},
+        run: vi.fn(okWithKnownGatewayList),
+        runDocker,
+      },
+    );
+
+    expect(result.exitCode).toBe(1);
+    expect(runDocker.mock.calls.some(([args]) => args[0] === "rm")).toBe(false);
+    expect(errors.join("\n")).toContain("remains after ownership-aware cleanup");
+  });
+
+  it("stops uninstall when orphaned host-local vLLM removal fails (#8981)", () => {
+    const errors: string[] = [];
+    const containerId = "c".repeat(64);
+    const runDocker = dockerResults(
+      new Map([
+        [JSON.stringify(ORPHANED_VLLM_INSPECT_ARGS), ok(`${containerId} true\n`)],
+        [JSON.stringify(["rm", "-f", containerId]), notFound()],
+      ]),
+    );
+
+    const result = runUninstallPlan(
+      { assumeYes: true, deleteModels: false, keepOpenShell: true },
+      {
+        commandExists: (command) => command === "openshell" || command === "docker",
+        env: { HOME: "/tmp/nemoclaw-uninstall-vllm-removal-failure" } as NodeJS.ProcessEnv,
+        existsSync: () => false,
+        error: (message) => errors.push(message),
+        isTty: false,
+        log: () => {},
+        run: vi.fn(okWithKnownGatewayList),
+        runDocker,
+      },
+    );
+
+    expect(result.exitCode).toBe(1);
+    expect(runDocker).toHaveBeenCalledWith(
+      ["rm", "-f", containerId],
+      expect.objectContaining({ timeout: 10_000 }),
+    );
+    expect(errors.join("\n")).toContain("Could not remove orphaned managed inference container");
+    expect(runDocker.mock.calls.some(([args]) => args[0] === "ps")).toBe(false);
+  });
+
   it("fails before generic Docker cleanup when a reserved inference name remains", () => {
     const errors: string[] = [];
     const psResults = new Map([
+      [
+        JSON.stringify([
+          "container",
+          "inspect",
+          "--format",
+          '{{.Id}} {{index .Config.Labels "com.nvidia.nemoclaw.managed-vllm"}}',
+          "nemoclaw-vllm",
+        ]),
+        notFound(),
+      ],
       [JSON.stringify(["ps", "-a", "--format", "{{.Names}}"]), ok("nemoclaw-llama-cpp\n")],
       [
         JSON.stringify(["ps", "-a", "--format", "{{.ID}} {{.Image}} {{.Names}}"]),

src/lib/actions/uninstall/run-plan.ts

@@ -30,6 +30,8 @@ import {
 import { buildUninstallPlan, type UninstallPlan } from "../../domain/uninstall/plan";
 import {
   cleanupManagedLlamaCppRuntimeForSandbox,
+  HOST_LOCAL_VLLM_CONTAINER_NAME,
+  HOST_LOCAL_VLLM_MANAGED_LABEL,
   type ManagedLlamaCppCleanupTarget,
   resolveManagedLlamaCppCleanupTarget,
 } from "../../inference/local-model-profile/cleanup";
@@ -1595,6 +1597,9 @@ function removeHostLocalModelRuntimes(paths: UninstallPaths, runtime: UninstallR
     DUAL_STATION_VLLM_RUNTIME_RECEIPT_FILE,
   ].some((name) => runtime.existsSync(path.join(sharedRoot, name)));
   if (!hasLlamaState && (!hasManagedKey || hasDistributedReceipt)) {
+    if (!hasManagedKey && !hasDistributedReceipt && !removeOrphanedManagedHostLocalVllm(runtime)) {
+      return false;
+    }
     return true;
   }
   const result = runtime.runLocalModelRuntimeCleanup({
@@ -1608,6 +1613,34 @@ function removeHostLocalModelRuntimes(paths: UninstallPaths, runtime: UninstallR
   return false;
 }
 
+function removeOrphanedManagedHostLocalVllm(runtime: UninstallRuntime): boolean {
+  if (!runtime.commandExists("docker")) return true;
+  const inspection = runtime.runDocker(
+    [
+      "container",
+      "inspect",
+      "--format",
+      `{{.Id}} {{index .Config.Labels ${JSON.stringify(HOST_LOCAL_VLLM_MANAGED_LABEL)}}}`,
+      HOST_LOCAL_VLLM_CONTAINER_NAME,
+    ],
+    { env: runtime.env, timeout: 10_000 },
+  );
+  if (inspection.status !== 0 || !inspection.stdout.trim()) return true;
+  const [containerId, managedLabel, ...extra] = inspection.stdout.trim().split(/\s+/);
+  if (!/^[0-9a-f]{64}$/u.test(containerId ?? "") || managedLabel !== "true" || extra.length > 0) {
+    return true;
+  }
+  const removal = runtime.runDocker(["rm", "-f", containerId], {
+    env: runtime.env,
+    timeout: 10_000,
+  });
+  if (removal.status === 0) return true;
+  runtime.error(
+    `Could not remove orphaned managed inference container '${HOST_LOCAL_VLLM_CONTAINER_NAME}'. NemoClaw did not start the remaining uninstall steps.`,
+  );
+  return false;
+}
+
 function managedLlamaCppCleanupPorts(homeDir: string, scopedToSelectedGateway: boolean): number[] {
   const ports = new Set<number>([GATEWAY_PORT]);
   if (scopedToSelectedGateway) return [...ports];

src/lib/inference/local-model-profile/cleanup.ts

@@ -46,6 +46,8 @@ import {
 } from "../serving/vllm-host-local-lifecycle";
 import { loadManagedVllmApiKey, managedVllmStateDir } from "../vllm-api-key";
 
+export { HOST_LOCAL_VLLM_CONTAINER_NAME, HOST_LOCAL_VLLM_MANAGED_LABEL };
+
 const LLAMA_MANAGED_LABEL = "io.nvidia.nemoclaw.host-local-inference.managed";
 const LLAMA_PROVIDER_LABEL = "io.nvidia.nemoclaw.host-local-inference.provider";
 const LLAMA_SERVICE_LABEL = "io.nvidia.nemoclaw.host-local-inference.service";

src/lib/onboard.ts

@@ -3502,6 +3502,7 @@ const {
   buildOrphanedSandboxRollbackMessage,
   ensureDashboardForward,
   ensureAgentDashboardForward,
+  ensureFinalizationDashboardForward,
   ensureAgentFixedForward,
   fetchGatewayAuthTokenFromSandbox,
   getDashboardForwardPort,
@@ -3522,7 +3523,6 @@ const {
   sleep: sleepSeconds,
   printAgentDashboardUi: agentOnboard.printDashboardUi,
 });
-
 const onboardRuntimeBoundary = new OnboardRuntimeBoundary({
   toSessionUpdates: (updates: Record<string, unknown>) =>
     toSessionUpdates(updates as Parameters<typeof toSessionUpdates>[0]),
@@ -4240,7 +4240,7 @@ async function runOnboard(opts: OnboardOptions = {}): Promise<void> {
         webSearchProvider: (config) => webSearchProviderForConfig(config),
       },
       finalizationDeps: {
-        ensureAgentDashboardForward: (name, selectedAgent) => selectedAgent ? ensureAgentDashboardForward(name, selectedAgent) : ensureDashboardForward(name, process.env.CHAT_UI_URL),
+        ensureAgentDashboardForward: (name, selectedAgent) => selectedAgent ? ensureAgentDashboardForward(name, selectedAgent) : ensureFinalizationDashboardForward(name),
         setDefaultSandbox: registry.setDefault,
         verifyWebSearchInsideSandbox,
         toSessionUpdates: (updates) =>

src/lib/onboard/agent-dashboard-forward.ts

@@ -117,7 +117,7 @@ export async function ensureAgentDashboardForward(options: {
   return actualAgentDashboardPort;
 }
 
-function replaceUrlPort(value: string, port: number): string {
+export function replaceUrlPort(value: string, port: number): string {
   try {
     const parsed = new URL(value.includes("://") ? value : `http://${value}`);
     parsed.port = String(port);