Node-control WS has write-only liveness: a blackholed connection strands the node offline for hours while /health reports connected

[khaliqgant]

Symptom

A fleet node silently leaves the roster and never comes back on its own. agent-relay fleet nodes hides it (it is offline, visible only under --all) while the broker process is alive and its own /health reports:

{"nodeConnected":true,"nodeDelivery":{"connected":true,"tokenPresent":true},"relaycastConnected":true,"status":"ok"}

lsof on the broker shows the TLS socket to the engine still ESTABLISHED. Nothing recovers it but a broker restart, and there is no /api/* route to force a node-control reconnect.

Incidence

finn-mini (node_d4190c4c2ca5c26bf547301347af4028, relay-broker/11.4.0), 2026-08-07. Engine-side lastHeartbeatAt froze at 11:52:35Z and did not move across repeated polls for ~80 minutes, while sf-mini on the same workspace heartbeat every ~30s. Broker uptime at the time was 26h, so this was not a startup failure.

Mechanism

crates/broker/src/node_control.rs — every disconnect path in run_connected_once's select! keys off send_wire(...).is_err():

_ = heartbeat.tick() => {
    if send_wire(&mut sink, &BrokerToRelaycast::NodeHeartbeat(...)).await.is_err() {
        return ControlRunResult::Disconnected;
    }
}
message = stream.next() => { ... }   // no timeout arm

That is write-only liveness. On a blackholed connection the kernel keeps accepting 12-second heartbeat frames into the send buffer, so send_wire never errors. There is no heartbeat ack, no WS ping/pong, and no read-side deadline — stream.next() waits forever. The client stays in the select! indefinitely, so the reconnect/backoff machinery that already exists below it is never reached.

The engine is not guaranteed to send unsolicited traffic, so an application-level ack is not sufficient on its own; the connection needs its own keepalive. #1450 notes the events WebSocket is already pinged every 30s — node-control is the socket that isn't.

Fix

Send a WS ping on each heartbeat tick and track the last inbound frame; if nothing arrives within 4 heartbeat intervals (48s), return ControlRunResult::Disconnected and let the existing backoff reconnect.

Related

• Hosted→broker inbound delivery dies after broker restart and stays dead through subsequent restarts — health reports the link as connected #1386 — same class of dishonest health reporting, but for hosted->broker delivery after restart. Its stated secondary defect (health should reflect end-to-end deliverability, not socket presence) applies to nodeConnected here too.
• broker: PTY input WebSocket has no keepalive while the events WebSocket is pinged every 30s #1450 — PTY input WebSocket has no keepalive; same root cause on a different socket.