fix(model-router): serialize lifecycle across gateways

docs/inference/set-up-model-router.mdx

@@ -32,9 +32,34 @@ When the log contains readable output from this run, the error includes a redact
 The owner-only log contains unredacted router output.
 Review the log before you share it.
 NemoClaw does not provide log-only cleanup.
-When `destroy` removes the last registered Model Router sandbox that uses a host port, it also stops the Model Router process and frees that port.
-If the stop fails, destroy still completes and prints a warning with the manual stop command.
-While another registered Model Router sandbox uses the same host port, destroying one Model Router sandbox keeps that Model Router process running.
+
+## Router and Sandbox Lifecycle Locks
+
+Routed onboarding already holds the onboarding session lock when it acquires the selected gateway route lock, then the current-user lock for the selected Model Router port.
+The port lock is shared across the current user's NemoClaw gateways.
+Onboarding holds all three locks through router setup and sandbox registry publication.
+Before sandbox deletion, `destroy` captures the current onboarding session identity.
+Model Router destruction takes the gateway route lock, then the current-user Model Router port lock, and then tries the onboarding session lock without waiting.
+After acquiring the session lock, `destroy` rechecks the captured identity before the peer check or process stop.
+If another onboarding run owns the session lock, or the identity changed, `destroy` skips Model Router teardown and warns.
+It leaves the Model Router process and current onboarding session unchanged.
+When `destroy` removes a Model Router sandbox, it checks the bounded set of NemoClaw gateway registries under the host state directory for a same-port peer.
+If no same-port peer remains, NemoClaw stops only a process whose command line still identifies the Model Router on that port.
+NemoClaw clears the matching Model Router process and credential recovery identity only after the stop succeeds or a complete process scan and health probe confirm that the router is absent.
+The final sandbox-name cleanup also checks the captured session identity.
+Other sandbox destroy paths use a non-blocking session update.
+If onboarding owns the lock or the captured session identity changed, `destroy` leaves the current session unchanged.
+If the captured session still names the destroyed sandbox but uses another router port, `destroy` clears only the sandbox association and preserves the router process and credential recovery identity.
+If the process inventory is unavailable, the completed scan finds no matching process while the port remains healthy, or the stop fails, `destroy` still completes and keeps the recovery identity.
+If `destroy` warns that it could not identify or stop a listener for the deleted sandbox, follow these steps:
+
+1. Inspect the current listener process immediately before you stop anything.
+2. Stop it only if its command line identifies the Model Router on the named port.
+3. Do not stop the router recorded by a preserved session for another port.
+4. Do not stop a previously reported process ID if its command line no longer matches.
+
+While another registered Model Router sandbox in any host gateway registry uses the same port, destroying one Model Router sandbox keeps the process running.
+A Model Router sandbox on another port does not keep the process running.
 A successful [uninstall](../../manage-sandboxes/operate-sandboxes/uninstall-nemoclaw) stops the selected Model Router and removes its log with the selected gateway's operational state.
 Review the uninstall scope before you use it for log removal.
 

docs/reference/commands.mdx

@@ -2301,7 +2301,17 @@ Model traffic uses the OpenShell-managed `inference.local` route configured by N
 Stop managed local inference resources, remove the host-side Docker image built during onboard, and delete the sandbox.
 This removes the sandbox from the registry.
 For Ollama-backed sandboxes, `destroy` also asks Ollama to unload currently loaded models and clears stale auth proxy state on a best-effort basis.
-When `destroy` removes the last registered Model Router sandbox that uses a host port, it also stops the host Model Router process on a best-effort basis and frees that port.
+For Model Router sandboxes, `destroy` keeps the process and recovery identity when another sandbox uses the port or when session, process, or absence checks are inconclusive.
+It also preserves a replacement onboarding session when the captured session identity changed.
+If the captured session uses the destroyed sandbox name with another router port, `destroy` clears only the sandbox association and preserves that router's recovery identity.
+For lock order, same-port peer handling, and cleanup checks, refer to [Set Up Model Router](../inference/hosted-inference/set-up-model-router#router-and-sandbox-lifecycle-locks).
+
+If `destroy` warns that it could not identify or stop a listener for the deleted sandbox:
+
+1. Inspect the current listener process immediately before you stop anything.
+2. Stop it only if its command line identifies the Model Router on the named port.
+3. Do not stop the router recorded by a preserved session for another port.
+4. Do not stop a previously reported process ID if its command line no longer matches.
 
 <Warning>
 This command attempts to wipe the manifest-defined agent state while its persistent volume is mounted, then removes the sandbox.

src/lib/actions/sandbox/destroy-flow.test.ts

@@ -64,6 +64,44 @@ describe("destroySandbox flow", () => {
     );
   });
 
+  it("runs routed teardown under the gateway and host router-port locks (#9098)", async () => {
+    const harness = createDestroyHarness({ provider: "nvidia-router" });
+
+    await expect(harness.destroySandbox("alpha", { yes: true })).resolves.toBeUndefined();
+
+    expect(harness.withGatewayRouteMutationLockSpy).toHaveBeenCalledWith(
+      "nemoclaw-19080",
+      expect.any(Function),
+    );
+    expect(harness.withModelRouterPortLifecycleLockSpy).toHaveBeenCalledWith(
+      4000,
+      expect.any(Function),
+    );
+  });
+
+  it("leaves an active same-name replacement onboarding session unchanged", async () => {
+    const harness = createDestroyHarness({
+      provider: "nvidia-router",
+      endpointUrl: "http://host.openshell.internal:4000/v1",
+      replaceSessionAfterRegistryRemoval: true,
+      sessionRouterPid: 4242,
+    });
+
+    await expect(harness.destroySandbox("alpha", { yes: true })).resolves.toBeUndefined();
+
+    expect(harness.stopModelRouterForDestroyedSandboxSpy).toHaveBeenCalledOnce();
+    expect(harness.compareAndSwapSessionSpy).not.toHaveBeenCalled();
+    expect(harness.updateSessionSpy).not.toHaveBeenCalled();
+    expect(harness.sessionState).toMatchObject({
+      sessionId: "replacement-session",
+      sandboxName: "alpha",
+      endpointUrl: "http://host.openshell.internal:4000/v1",
+      routerPid: 6262,
+      routerCredentialHash: "replacement-hash",
+    });
+    expect(harness.warnSpy).toHaveBeenCalledWith(expect.stringContaining("owns the session lock"));
+  });
+
   it("revokes the prior HTTPS-pin route only after confirmed deletion and registry removal", async () => {
     const routeId = "a".repeat(64);
     const harness = createDestroyHarness({
@@ -476,7 +514,8 @@ describe("destroySandbox flow", () => {
       timeout: 30_000,
     });
     expect(harness.removeSandboxSpy).toHaveBeenCalledWith("alpha");
-    expect(harness.updateSessionSpy).toHaveBeenCalledOnce();
+    expect(harness.compareAndSwapSessionSpy).toHaveBeenCalledOnce();
+    expect(harness.updateSessionSpy).not.toHaveBeenCalled();
     expect(harness.logSpy.mock.calls.map((call) => String(call[0])).join("\n")).toContain(
       "Sandbox 'alpha' destroyed",
     );
@@ -725,7 +764,8 @@ describe("destroySandbox flow", () => {
     });
     expect(harness.finalizeMcpBridgesAfterSandboxDeleteSpy).toHaveBeenCalledTimes(2);
     expect(harness.removeSandboxSpy).toHaveBeenCalledWith("alpha");
-    expect(harness.updateSessionSpy).toHaveBeenCalledOnce();
+    expect(harness.compareAndSwapSessionSpy).toHaveBeenCalledOnce();
+    expect(harness.updateSessionSpy).not.toHaveBeenCalled();
     expect(harness.cleanupGatewaySpy).toHaveBeenCalledWith(
       "nemoclaw-19080",
       harness.runOpenshellSpy,