Investigate why agent-relay brokers end up SIGSTOP'd holding the SQLite write lock

[khaliqgant]

Problem

Two agent-relay processes were found in state T/Ts (stopped), one having been suspended since Jul 30 and the other since Aug 1:

PID    STARTED                  ELAPSED       STAT  COMMAND
13214  Thu Jul 30 21:48:37      03-10:55:06   T     agent-relay node up
85652  Sat Aug  1 19:22:19      01-13:21:24   Ts    agent-relay node up

One was frozen mid-write-transaction, so it held the SQLite write lock permanently. ai-history.db and its WAL were frozen at Aug 2 21:37, and every ai-hist sync failed with database is locked in 0s for days.

Questions

• What suspends these processes? SIGSTOP from a supervisor/watchdog, Ctrl-Z from an interactive shell, a debugger attach, or macOS App Nap / process throttling?
• Why do two long-lived brokers hold the database open continuously at all, rather than opening for the duration of a write and closing?
• Should the broker hold a write transaction across any operation that can block?

Mitigations regardless of cause

• Bounded write transactions in the broker: never hold one across I/O or idle time
• Better: stop writing SQLite from the broker entirely (see Route all SQLite writers through append-only spools (single-writer architecture) #47)
• Detection: feat(ai-hist): read-only reads, lock diagnostics, and write guardrails #46's ai-hist doctor now flags stopped/zombie holders and prints kill -CONT <pid>

Note the resolution is to resume these processes, not stop them — they are already stopped, and resuming lets the stuck transaction finish and release the lock.

Related: #46, #47

[kjgbot]

Investigation verdict, coordinated with the Relay-side agent-silent-death-incident lane:

SQLite did not cause the reported 2026-08-06T13:00:19.000Z fleet freeze. That timestamp was one inventory.sync reconciliation time applied to 25 agents.lastSeen rows after a transport reconnect, not an observed process-stop time. No PIDs or process states were captured for that cohort; PID 23630 had TLS activity across the timestamp, and direct probes subsequently advanced the sampled lanes.

The historical T/Ts brokers in this issue are real but separate. The supported causal direction is external process stop -> possibly frozen in-flight SQLite transaction. SQLite lock waits return SQLITE_BUSY/SQLITE_LOCKED; they do not deliver SIGSTOP or block Relay's Node event loop because the N-API call runs on a blocking worker. The original sender/reason for the Jul 30/Aug 1 stop signals is not recoverable from the retained evidence, so attributing it to a supervisor, terminal job control, or operator would be speculation.

Draft PR #59 records the full evidence and implements the diagnosis-supported mitigations: bounded busy retries, per-source isolation, stale temp cleanup, and automatic fresh write-capability diagnostics. It does not claim those mitigations fix Relay fleet liveness.