Redesigning ProcessMonitor: detect exit from the child, not its pipes

[SeanTAllen]

ProcessMonitor runs a child process and reports two things back: the child's output, and its exit status once the child exits. The problem is the exit status. The monitor often can't detect that the child has exited, so it hangs, leaves the child as a zombie, and never reports the status.

The monitor never detects the child's exit directly. It relies on the pipes: when the child's stdout and stderr both reach end-of-file, the monitor reports the child as exited. But end-of-file on a pipe means only that every handle on the write end has been closed, and the child is not always the last process holding that write end open. A grandchild the child spawned can inherit that write end. The parent can still hold the child's stdin open. While something else holds the pipe open, it stays open after the child exits; the EOF never arrives, and the monitor never reports exit. This one wrong assumption causes both #5764 and #5748.

The fix is to detect the child's actual exit, delivered as a native per-platform event, and treat that as the only signal that the child has exited. Separately, a factory does the fallible setup before the actor exists and returns either a live monitor or an error, so the monitor is only ever created around a running child and can't be in a "failed to start" state at all. The redesign also fixes two related bugs: a kill-after-reap race (#5765) and a file-descriptor leak on fork failure (#5766).

This is larger than a bug fix. It changes the construction API, the exit-detection mechanism, and the actor's lifecycle. It lands as a single cutover, built as reviewable sections on one branch.

The issues this addresses

• ProcessMonitor never reports a child's exit status if a grandchild keeps stdout or stderr open #5764: the child exits, but a grandchild that inherited stdout/stderr outlives it and holds the pipes open (sh -c "some_daemon & exit 7"). The pipes don't reach EOF until the grandchild exits, so dispose is delayed until then — or never fires, for a grandchild that outlives the parent (a daemon) — and the child stays a zombie in the meantime. This is the case that motivated the redesign.
• ProcessMonitor never reports a child's exit status if stdin is still open #5748: the exit status is not reported while the parent holds the child's stdin open. The same fault, fixed by the same change. (An earlier interim fix, PR Fix ProcessMonitor never reporting a child's exit status when stdin is open #5763, was closed in favor of fixing ProcessMonitor never reports a child's exit status if stdin is still open #5748 as part of this redesign.)
• ProcessMonitor.dispose can signal an unrelated process after the child has been reaped #5765 (POSIX): dispose() calls kill() on a pid that may already have been reaped and recycled. The redesign makes the guard against this structural.
• ProcessMonitor leaks pipes and backpressure when fork() fails #5766: the fork-error path leaks the pipes it opened. The factory's cleanup path closes them.

Divergences from today's behavior

The first item is the fix; the rest are behavior or API changes. "Today" means released ponyc (main).

1. Fixed: the grandchild case (ProcessMonitor never reports a child's exit status if a grandchild keeps stdout or stderr open #5764) now reports dispose(Exited(7)) promptly.
2. API change (construction): ProcessMonitor(...) no longer always returns a monitor. Construction becomes a factory that returns either a ProcessMonitor or a ProcessError. Preconditions that were reported asynchronously through ProcessNotify.failed (no exec capability, missing file, and now an unsupported kernel) are returned synchronously instead, and no monitor is created for them.
3. Behavior change (descendant output): output that a descendant writes to the inherited stdout/stderr after the child exits is no longer delivered; it is cut off at the reap. Today it is delivered in full, or the monitor hangs (that hang is ProcessMonitor never reports a child's exit status if a grandchild keeps stdout or stderr open #5764).
4. Behavior change (pipe errors): a read or write error (POLLERR) on one pipe now closes only that pipe, and the child keeps being monitored to its actual exit, instead of the monitor tearing down.
5. Platform requirement: exit detection uses a native per-platform event. On Linux that needs pidfd_open (kernel ≥ 5.3); on an older kernel, construction returns a factory error instead of failing silently by never disposing. macOS, the BSDs (EVFILT_PROC), and Windows (a waitable process handle) have no version floor.

The ProcessNotify callback contract is otherwise unchanged, and "all of the child's own output is delivered before dispose" is preserved, with two narrow exceptions listed under Out of scope.

The design

Construction: a supply-chain factory

Actor constructors can't fail, so the current constructor does all the fallible setup (capability check, file check, pipe creation, fork) inside the actor and reports failures asynchronously through ProcessNotify.failed, because a constructor has no other way to signal. The supply-chain pattern moves that work before the actor exists:

match StartProcess(sp_auth, bp_auth, notifier, path, args, vars)
| let pm: ProcessMonitor => // a live child is running
| let err: ProcessError  => // never started; here's why, synchronously
end

The factory (name provisional, see Decisions taken) does every fallible step: validate the exec capability, check that the file exists, probe kernel support (pidfd_open on our own pid, where ENOSYS means the kernel is too old), create the four pipes, and fork/exec. On any synchronous failure it closes whatever it opened and returns a ProcessError. On success it builds the actor around the live child and the pipes.

The pipes and the child cross into the actor as iso and are consumed. The actor's constructor calls begin() on each pipe, registering its asio event with the actor as owner (the same call today's constructor already makes), registers the exit event, and fires created. A compile probe that models this pattern — a factory returning a monitor-or-error union, with iso resources consumed into an actor that owns their asio events — compiles, so the pattern is feasible in Pony's type system.

So the capability, missing-file, unsupported-kernel, and synchronous-fork failures are factory errors with no actor. The genuinely asynchronous failures — execve or chdir failing in the forked child, surfaced later through the _err relay pipe — remain events that the running actor reports through ProcessNotify.failed, exactly as today.

Exit detection: a native per-platform event

The child's exit is delivered as an asio event — the same machinery the pipes use — not a poll. When it fires, the actor reaps the child (waitpid, the Windows handle wait, or GetExitCodeProcess) to collect the status and clear the zombie, then runs the reap edge (the atomic teardown-and-report sequence defined below). The exit event is the only signal of exit. Pipe-EOF only means "stop reading this pipe," and a read or write error on a pipe closes only that pipe: a pipe-level problem is not a child exit.

Per platform:

• Linux, pidfd_open(): the pidfd goes readable when the child exits and registers with epoll like any other fd. Pony's epoll backend already delivers arbitrary fds through the read-event path with no runtime C change. A pidfd refers to the exact process, so it is immune to pid recycling, which strengthens the ProcessMonitor.dispose can signal an unrelated process after the child has been reaped #5765 guarantee rather than relying on it.
• macOS, FreeBSD, DragonFly, OpenBSD — kqueue EVFILT_PROC + NOTE_EXIT: registers with the existing kqueue poller. This one needs a contained backend extension, because kqueue keys on a pid, not an fd, so there is no arbitrary-fd path today.
• Windows, the process handle: a waitable handle that signals on exit, bridged to the existing completion port via RegisterWaitForSingleObject — the same pattern the console-stdin path already uses. Windows also polls its pipe reads on a timer today (PeekNamedPipe), with no asio wired for pipes there, so on Windows we move both the exit event and the pipe reads onto asio. That is the substantial part of the platform's change.

The per-child exit poll is gone on every platform, and once the Windows pipe reads are on asio there is no poll timer left anywhere. A polling design would have needed a per-child Timers actor, which allocates eagerly; with no timer, that cost is gone too.

State model: three states

Because the factory guarantees a live child before the actor exists, the actor has no "not started" and no "failed to start"; its existence means a child is running. The lifecycle flags collapse to one named, exhaustively-matched union:

primitive _Running
primitive _Disposing
primitive _Reaped
type _Lifecycle is (_Running | _Disposing | _Reaped)

• _Running: child alive, being monitored. Reads deliver stdout/stderr, writes queue to stdin with backpressure, exec/chdir errors surface via the _err pipe, and the exit event is armed.
• _Disposing: the client called dispose(), so we killed the child, closed the pipes, and are awaiting the exit event. Distinct from _Running because the reap that follows must not deliver more output (the client cancelled), where a natural-exit reap does, and because a second dispose() here is a no-op rather than a redundant kill().
• _Reaped: the exit event fired and we collected the status (or a reap error); dispose/failed fired; absorbing and inert. dispose() here is a no-op and never calls kill(). It still destroys asio events on the disposable callback, so nothing leaks and the actor can be collected.

The factory makes "the actor exists only if a child is running" true, and the type system enforces it. A trait-based state-object machine was rejected: the pipes, read buffer, and pending-write list are per-actor and shared across _Running and the reap, so a state-object split would not help.

Transitions

Each invariant-bearing action happens on exactly one edge.

From	Event	To	Actions
(factory success)	—	_Running	actor created with live child + pipes; begin() each pipe, arm the exit event, notifier.created
_Running	exit event → reap = status	_Reaped	the reap edge (below): go _Reaped, drain, close all, notifier.dispose(status)
_Running	exit event → reap errors	_Reaped	close all pipes, notifier.failed(WaitpidError) (no dispose)
_Running	client dispose()	_Disposing	_child.kill(), close all pipes (releases backpressure); exit event stays armed
_Disposing	exit event → reap = status	_Reaped	notifier.dispose(status), no drain (pipes already closed; client cancelled)
_Disposing	exit event → reap errors	_Reaped	pipes already closed; notifier.failed(WaitpidError) (no dispose)
_Disposing	client dispose()	_Disposing	no-op, do not kill() again
_Reaped	client dispose()	_Reaped	no-op, never kill() (#5765)

notifier.dispose(status) runs only on a status-carrying edge into _Reaped; those edges are mutually exclusive and _Reaped is absorbing, so dispose fires exactly once, by construction.

The reap edge

When the exit event fires and the reap returns a status, the edge is atomic: one behavior turn, no yield in the middle.

1. set _Reaped (the status is a local threaded to step 4; _Reaped holds no payload);
2. drain stdout, stderr, and _err with a dedicated synchronous, non-yielding loop (not the 4 KB-yielding _pending_reads path, which would either truncate the drain or break the edge's atomicity), reading each to EAGAIN/EOF, bounded by a cap;
3. close all four pipes (both ends, idempotent), releasing backpressure;
4. notifier.dispose(status).

Draining preserves "all of the child's own output is delivered before dispose." When the exit event fires, the child's last writes may still sit unread in the pipe buffer. A dead child can't still be writing, and its writes reached the buffer before it exited, so the unread remainder is bounded by one pipe capacity. Draining _err here is necessary: on an exec/chdir failure the child writes one byte to _err and exits, so the reap must read that byte and fire notifier.failed(...) before closing _err, or the failure report is silently dropped.

The drain is capped so that a descendant flooding the pipe the instant the child dies can't stall teardown. The cap must be at least one pipe capacity, and it must be a fixed constant or bounded ceiling, not a function of the pipe's buffer size, because the child can grow that buffer (Linux F_SETPIPE_SZ, up to 1 MiB) and inflate the per-reap work. It is a security bound, not a performance tuning parameter.

Pipe lifecycle

The factory creates the four pipes. begin() runs in the actor after it is created, creating the asio event and closing the far end.

• stdin closes only on client action — done_writing() (once pending writes drain), client dispose(), or a hard write error — plus teardown cleanup. It is never closed as a side effect of stdout/stderr reaching EOF. The client owns stdin's lifetime.
• stdout, stderr, err close on their own read-EOF while running, or at teardown even if EOF never came.

Every running-state stdin close routes through one _close_stdin() helper (close the near fd, release the backpressure flag, clear the pending queue), so backpressure can never be stranded. Teardown closes all four pipes with _Pipe.close() (both ends).

Invariants

• dispose fires exactly once: one status-carrying edge into the absorbing _Reaped, as derived under Transitions.
• Backpressure applied is always released: every stdin close goes through _close_stdin() (see Pipe lifecycle), so no path abandons stdin without releasing.
• kill() never targets a reaped pid (ProcessMonitor.dispose can signal an unrelated process after the child has been reaped #5765): kill() is reachable only on the _Running → _Disposing transition; _Disposing and _Reaped never call it. At that point the child is an unreaped zombie we own, so its pid can't be recycled. On Linux the pidfd refers to the exact process regardless, and Windows waits on a handle that never recycles.
• No fd or asio-event leak (ProcessMonitor leaks pipes and backpressure when fork() fails #5766): every factory failure path closes both ends of every opened pipe, every terminal path closes all four, and asio events are unsubscribed at close and destroyed on the disposable callback.

Error vocabulary

The factory return type carries construction failures as data. Today ExecveError means two different things: "the file doesn't exist" (a precondition, checked in the constructor) and "execve failed in the child" (an async failure, via the _err relay). The redesign gives the precondition its own variant, so missing-file becomes a distinct error type returned synchronously by the factory, and leaves ExecveError for the actual execve failure. The two meanings are now distinct types, not just distinct delivery channels. A variant is also added for an unsupported kernel. Moving the construction failures to the factory also removes a major source of the failed callback's double meaning; what remains on failed is the operational async errors (exec/chdir relay, write errors) plus a terminal reap error.

Latent bugs this redesign also fixes

• Backpressure leak on a hard write error. In _pending_writes, which runs only while the pending queue is non-empty (so backpressure is applied), the non-EAGAIN write-error branch closes stdin without releasing backpressure, stranding the client. Routing every stdin close through _close_stdin() fixes it. (The matching branch in _write_final runs only with an empty pending queue, where no backpressure is applied, so it doesn't strand.)
• _err fd/event leak on teardown. Today's _close() closes stdin, stdout, and stderr, but never _err. Closing all four on teardown fixes it, and it is also what makes draining _err on the reap edge necessary (see The reap edge).

Decisions taken (open to discussion feedback)

These were settled deliberately while writing the plan. Push back on any of them in the comments:

• Descendant-output truncation (Divergence 3), accepted. A descendant that outlives the child has its post-exit output dropped at the reap. This follows from monitoring the child rather than its descendants, and following descendants is exactly ProcessMonitor never reports a child's exit status if a grandchild keeps stdout or stderr open #5764's hang: there is no design that both fixes the bug and keeps delivering the descendant's output. Revisit if a real use case surfaces.
• Rollout, release-note-only. The API change won't compile until callers update it, so nobody misses it, and the behavior changes go in the release notes. No consumer survey or compatibility shim beyond this discussion, which is itself the advance notice to the community.
• Terminal states carry no payload. _Reaped is a plain marker: the exit status goes out through dispose(status) at the transition and isn't read afterward, so the state doesn't need to hold it (unlike the factory's error, which is a return value).
• Testing ProcessMonitor.dispose can signal an unrelated process after the child has been reaped #5765 via an injectable _Process seam, adopted. See Testing.
• Names are provisional (StartProcess, _Running/_Disposing/_Reaped, the unsupported-kernel error). Renaming is cheap and separable from the design.

Implementation notes

• The kqueue and Windows work. Exit detection above covers each platform's mechanism and cost. Two implementation specifics not stated there: the kqueue EVFILT_PROC arm needs a shared asio flag (the backend keys its other filters on an fd), and the Windows bridge has to reconcile process-handle ownership with the reap (the current Windows wait closes the handle).
• Fork/pidfd race. Plain fork then pidfd_open, or clone3 with CLONE_PIDFD to get the pidfd atomically at fork. Both are safe (the child's pid can't be recycled before we reap); an implementation choice, not a blocker.
• The reap call. Even though the event signals that the child exited, we still call waitpid to collect the status and reap. Keep it non-blocking so it can't hold a scheduler thread, and pin the WNOHANG-without-WUNTRACED assumption with a comment (a stopped child would otherwise be counted as exited).
• _ProcessNone and its fabricated Exited(255) largely disappear: with the factory, the actor is never created without a real child.

Testing

The key trap: asserting that dispose(Exited(7)) fires is not a discriminating test for #5764. On buggy code, when the grandchild eventually releases the pipes, the same status is reported, just late or never. A status-only assertion passes even on the bug. The only thing that separates correct from broken is time, so every #5764 and stdin test must be built around a categorical time separation (grandchild outlives the timeout, an unbounded self-terminating grandchild, a wide elapsed window), not a tuned latency threshold. The existing suite's discipline carries over: assert on the eventual observable outcome, use a generous long_test timeout as the hang detector, serialize with exclusion_group, and prove negatives with a bounded timer window.

New tests: the silent grandchild (sh -c "sleep 30 & exit 7", timeout well under the grandchild's life); the grandchild case delivering the child's buffered output; a flooding grandchild not stalling teardown (drain cap); a child that closes its output and blocks on stdin, completing only after a delayed done_writing(); dispose-exactly-once against a reap race; and a coarse fd-leak stress loop. The factory also makes the precondition failures (capability, missing file, unsupported kernel) synchronous and directly assertable, with no notifier round-trip.

The injectable _Process seam is the primary way to verify the state machine, and it is what makes generative testing possible. The factory gives the actor its child as a value, so a package-private test factory can substitute a spy _Process that records kill() and returns a chosen status from wait() on demand, with no public surface and no real process. On top of it, a PonyCheck property generates random orderings of the client-facing events (write, done_writing, dispose, "child exits now") and asserts the invariants hold for every ordering: dispose exactly once, kill() never after a reap (#5765), backpressure always released. This exercises the message-order races where the concurrency bugs occur (dispose-vs-natural-exit, write-after-exit, double-dispose), and since each behavior is atomic (the reap edge is one synchronous turn), message order is the whole race space. It does not fake pipe bytes: output delivery and the OS-level behavior (fd inheritance, a grandchild holding the pipes) stay with the real-process tests above. Two layers, then: property-based tests over the state machine, and real-process tests for integration.

Regression set to keep green, each pinning a contract clause: the exec/chdir-failure path fires both failed and dispose (strengthen these to assert failed, given the _err drop risk); dispose never fires when construction fails (now a factory error, so assert the error return); kill-while-running yields Signaled; and the stdin/stdout, ordering, expect, and #5748 tests.

Scope and sequencing

One branch, event-based end state, no interim polling release, landed as a cutover. Reviewable sections:

• A: the platform-agnostic core — the supply-chain factory, the three-state actor, the reap edge and drain, and the error-vocabulary cleanup. Paired with one exit trigger, this fixes ProcessMonitor never reports a child's exit status if a grandchild keeps stdout or stderr open #5764, ProcessMonitor.dispose can signal an unrelated process after the child has been reaped #5765, and ProcessMonitor leaks pipes and backpressure when fork() fails #5766.
• B: the Linux pidfd exit event (Pony-only).
• C: the kqueue EVFILT_PROC backend extension and wiring.
• D: the Windows RegisterWaitForSingleObject bridge, wiring, and handle ownership.
• E: tests.

There is no interim fix in released ponyc (PR #5763 was closed), so #5748 stays open until this lands, alongside #5764 and #5765.

Out of scope

• The failed callback still carries two async meanings (an operational error, with dispose to follow, and a terminal reap error). The factory removes the construction-failure case; the remaining split is pre-existing and preserved.
• With expect(qty) set, a final sub-qty residual at EOF is not delivered — a pre-existing exception to "all of the child's output is delivered."
• A child that grows its own pipe buffer past the drain cap and fills it before exiting has the overflow truncated — the adversarial exception to the same guarantee. The cap is a deliberate teardown bound.

[SeanTAllen]

Implemented.