Spawned agents produce nothing and the control plane cannot say whether they ran

[khaliqgant]

Corrected 2026-08-08 21:45Z. The first version of this issue claimed the codex spawn path fails to start a process, citing the local process table and lastSeen == createdAt. Both claims were wrong and are retained here as falsified rather than deleted. What survives is a real and arguably worse problem: the control plane cannot answer whether a spawned agent ran.

What was actually observed

Four agents dispatched at 21:25Z with cli: "codex", one per broker (sf-mini, finn-mini, barry, chief-broker). All four returned {"spawned": true} and registered. Six minutes later all four read status: offline, and none produced any work product — no branch, no comment, no message, on tasks that begin with a GraphQL query.

Four more dispatched at 21:32Z with cli: "claude". Three read status: active five minutes later.

Why the obvious conclusions do not hold

"lastSeen == createdAt proves it never ran" — false. At 21:37Z three healthy claude agents showed exactly that pattern while booting. For broker-spawned agents lastSeen appears to track messaging activity, not a timer, so a freshly spawned agent that has not yet spoken is indistinguishable from one that never started.

"No local process existed, so none was created" — false. cloud-2917-recovery-3 (chief-broker, cli: "codex", created 20:13:10Z) is alive and actively posting checkpoints, and it has no local codex process at its creation time either. Whatever the process topology is, the local table does not map to broker agents the way I assumed, so it cannot be used to prove absence.

status is not reliable either. soc2-hole1-review has a confirmed live process (pid 54737, state Ss+, started 4s after dispatch) and is absent from list_agents(status: "online"). Separately and repeatedly, the registry reports chief-khaliq offline while it is running and serving.

The actual defect

There is no trustworthy way to answer "did this agent ever run?"

• spawned: true describes a record write, not a running process.
• status produces false negatives on agents with live processes.
• lastSeen conflates "never started" with "started and hasn't spoken yet".
• The process table does not correspond to broker agents in a checkable way.

The practical cost tonight: four review lanes sat assigned for seven minutes, an operator could not distinguish booting from dead from never started, and the only recovery available was to re-dispatch blindly — which is how a name gets burned and how the same work gets done twice.

Two failures that need opposite responses are rendered identically. A dead agent lost work mid-flight; re-dispatch expecting partial progress. An agent that never started attempted nothing; re-dispatch expecting none. Today the caller cannot tell.

Asks

1. A liveness signal that dies with the process rather than a field written onto the record at registration. Registration fields are re-asserted and never revoked; they cannot express "gone".
2. Distinguish starting from running from exited, and make exited carry a reason. If spawn is asynchronous, spawned: true should resolve to one of those rather than terminate the caller's visibility.
3. Surface launch failure. Whatever happened to the four 21:25Z agents produced no error on any surface a caller can read.

Environment

Brokers: chief-broker 11.4.2, sf-mini 11.4.0, finn-mini 11.4.0, barry 11.3.1, all uncapped. Observed 2026-08-08 21:25–21:45Z.

[khaliqgant]

Resolved by direct observation: the agents are idle, not dead — the spawn prompt does not reach them

A spawned agent boots, registers, and then sits at 0% CPU with its brief never delivered. A later DM starts it.

The measurement

soc2-hole1-review, spawned 21:32:08Z on chief-broker with cli: "claude". Process pid 54737 started 4 seconds later and stayed up.

observed at	%CPU	RSS	state
21:44Z (12 min after spawn, before any DM)	0.0	103 MB	Ss+
21:52Z (~90s after a DM was sent)	18.8	238 MB	Ss+

Its CWD was /…/AgentWorkforce/chief the entire time — it had never cloned the repository its brief names, because it never received the brief. Ss+ is a session leader in the foreground process group: a harness parked on input.

The task it was given begins with a GraphQL query. Twelve minutes of 0.0% CPU is not slow progress; it is no progress.

What this explains

Three separate symptoms collapse into one cause:

• "Agents keep dying." They largely are not. They are alive, registered, and idle, having never been told what to do. An idle agent and a dead one produce identical output — none.
• Lanes that "died holding pushed work". Work pushed by a previous agent survived; the replacement never started, so the branch simply stopped moving.
• Re-dispatch not helping. Each new spawn inherits the same silence, which is what makes it look like a recurring death rather than a delivery failure.

Also: injectionMode: "steer" is silently downgraded

send_dm accepts injectionMode: "steer" and the returned message record reads "injectionMode": "wait". The parameter is accepted by the schema and not honoured by the running build, with no error — so the one documented escape hatch for reaching a parked agent cannot be selected by a caller. This sits directly alongside #1467/#1468, where wait is the mode that queues and can strand.

Revised asks

1. The spawn prompt must be delivered, or the spawn must fail. An agent that comes up without its brief is worse than one that fails to come up: it holds a name, occupies a slot, reports active, and does nothing.
2. Make "has this agent received work?" observable. Neither status, lastSeen, nor spawned: true distinguishes parked, never briefed from working.
3. Honour injectionMode, or reject it. Silently substituting wait for steer removes the caller's only lever precisely when it is needed.

Workaround for anyone hitting this now

After spawning, send the brief again as a DM. That is what moved 54737 from 0.0% to 18.8% CPU. It is not a fix — it doubles every dispatch and depends on a DM path that #1467 shows can itself strand — but it is what works today.

---

Supersedes the process-table reasoning in the corrected issue body above. That reasoning was wrong about mechanism; this is a direct before/after measurement on a single named process.