A node provider that dies without a graceful stop stays registered on the node and shadows its capabilities indefinitely. There is no supported way for an operator to see that it happened, and no supported way to undo it. Both halves are required — either one alone leaves the operator stuck.
What happened
A Herdr plugin attached a capability provider to a live local broker node (chief-broker) advertising spawn:claude / spawn:codex. Its host pane was closed, which killed the process without a graceful shutdown, so no node.deregister was ever sent.
The registration survived the process. Every subsequent placement onto that node was answered:
Provider "herdr-khaliqs-macbook-pro" is offline for action "spawn:claude"
Native spawning was blocked host-wide on a node with nine live agents on it, until the provider was cleared by hand. The teardown check at the time asserted the local process was gone — and passed, while the outage was active.
Half 1 — no operator-facing deregister
The primitive exists and works: NodeProviderClient.stop() is documented "Gracefully deregister the provider and close the connection" and emits { type: 'node.deregister', provider: { name, instance_id } }.
Nothing operator-facing calls it:
agent-relay fleet has no deregister subcommand — nodes, spawn, release, config, enable, disable, inherit, status. fleet release releases an agent, not a provider.grep -rn deregister packages/cli/src/ returns nothing. The only match in the CLI is a protocol schema literal in the bundled dist.
Recovery therefore required reaching past the CLI and scripting against the SDK: serveNode under the stranded provider's exact name (a fresh instance id replaces the previous attachment, per openSocket()'s documented reconnect-vs-duplicate arbitration), then abort its signal so stop() fires. That works, but it is not something an operator hitting this at 2am should have to derive from SDK source.
Two sharp edges anyone reproducing it must know, which is itself an argument for a first-class command:
nameOverride must be pinned to the node's existing name. node.register carries name alongside node_id, so attaching under any other name renames the live node for every consumer.- The replacement must register a harmless capability, never the contested
spawn:<cli>, or it can intercept a placement during the moments it is attached.
Suggested: agent-relay fleet provider release <name> (or similar), doing the re-attach + graceful stop, with the name pinning handled internally.
Half 2 — no way to enumerate providers
Nothing exposes providers, so the stranded one is invisible until it steals traffic:
agent-relay fleet nodes --all and query_nodes return only deduped capabilities with kind capacity. The provider's action capability collided by name with the broker's native capacity and never appeared.agent-relay fleet status reports "this node's provider attachment" but showed the same deduped capacity list.
The provider was invisible in all of them while attached, while stranded, and after removal — three different states, identical output. The only signal that distinguished them was issuing a real placement and seeing whether it was intercepted.
This is why the fix cannot be a deregister command alone. Without enumeration an operator cannot answer "is anything shadowing this node, and what is it called?" — and the deregister command needs a name as input. Conversely, enumeration without deregistration just lets you watch the problem.
Suggested: include attached providers (name, instance id, capabilities, last seen, live/stale) in the node record or a fleet providers command, and distinguish a provider-served capability from native capacity rather than deduping them into one entry.
Related
Worth considering whether the engine should expire a provider attachment that has missed heartbeats, rather than holding a dead registration that fails every placement routed to it. Failing closed on a provider known to be offline — instead of shadowing the node's working native capacity with it — would have prevented the outage regardless of tooling.
Environment: agent-relay 11.4.2, broker 11.4.2, protocol 2, macOS.
A node provider that dies without a graceful stop stays registered on the node and shadows its capabilities indefinitely. There is no supported way for an operator to see that it happened, and no supported way to undo it. Both halves are required — either one alone leaves the operator stuck.
What happened
A Herdr plugin attached a capability provider to a live local broker node (
chief-broker) advertisingspawn:claude/spawn:codex. Its host pane was closed, which killed the process without a graceful shutdown, so nonode.deregisterwas ever sent.The registration survived the process. Every subsequent placement onto that node was answered:
Native spawning was blocked host-wide on a node with nine live agents on it, until the provider was cleared by hand. The teardown check at the time asserted the local process was gone — and passed, while the outage was active.
Half 1 — no operator-facing deregister
The primitive exists and works:
NodeProviderClient.stop()is documented "Gracefully deregister the provider and close the connection" and emits{ type: 'node.deregister', provider: { name, instance_id } }.Nothing operator-facing calls it:
agent-relay fleethas no deregister subcommand —nodes,spawn,release,config,enable,disable,inherit,status.fleet releasereleases an agent, not a provider.grep -rn deregister packages/cli/src/returns nothing. The only match in the CLI is a protocol schema literal in the bundled dist.Recovery therefore required reaching past the CLI and scripting against the SDK:
serveNodeunder the stranded provider's exact name (a fresh instance id replaces the previous attachment, peropenSocket()'s documented reconnect-vs-duplicate arbitration), then abort its signal sostop()fires. That works, but it is not something an operator hitting this at 2am should have to derive from SDK source.Two sharp edges anyone reproducing it must know, which is itself an argument for a first-class command:
nameOverridemust be pinned to the node's existing name.node.registercarriesnamealongsidenode_id, so attaching under any other name renames the live node for every consumer.spawn:<cli>, or it can intercept a placement during the moments it is attached.Suggested:
agent-relay fleet provider release <name>(or similar), doing the re-attach + graceful stop, with the name pinning handled internally.Half 2 — no way to enumerate providers
Nothing exposes providers, so the stranded one is invisible until it steals traffic:
agent-relay fleet nodes --allandquery_nodesreturn only deduped capabilities with kindcapacity. The provider'sactioncapability collided by name with the broker's native capacity and never appeared.agent-relay fleet statusreports "this node's provider attachment" but showed the same deduped capacity list.The provider was invisible in all of them while attached, while stranded, and after removal — three different states, identical output. The only signal that distinguished them was issuing a real placement and seeing whether it was intercepted.
This is why the fix cannot be a deregister command alone. Without enumeration an operator cannot answer "is anything shadowing this node, and what is it called?" — and the deregister command needs a name as input. Conversely, enumeration without deregistration just lets you watch the problem.
Suggested: include attached providers (name, instance id, capabilities, last seen, live/stale) in the node record or a
fleet providerscommand, and distinguish a provider-served capability from native capacity rather than deduping them into one entry.Related
Worth considering whether the engine should expire a provider attachment that has missed heartbeats, rather than holding a dead registration that fails every placement routed to it. Failing closed on a provider known to be offline — instead of shadowing the node's working native capacity with it — would have prevented the outage regardless of tooling.
Environment: agent-relay 11.4.2, broker 11.4.2, protocol 2, macOS.