Skip to content

Commit 8c75fd1

Browse files
phil-oppclaude
andauthored
test(daemon): wait for the zombie with waitid, not a sysinfo poll (#3231)
* test(daemon): wait for the zombie with waitid, not a sysinfo poll `spawn_unreaped_wrapper` polled sysinfo for `ProcessStatus::Zombie` with a 5s deadline. That made the precondition depend on sysinfo *observing* the zombie, which is a platform question rather than the one under test: the poll timed out on the macOS nightly (#3150) while passing on Linux, failing both tests before they reached an assertion. `waitid(P_PID, pid, WEXITED | WNOWAIT)` answers the actual question — has it exited — and `WNOWAIT` leaves the status pending so it stays a zombie for the assertions and for `kill_group_and_reap`. Deterministic, no deadline. The sysinfo check is kept as an explicit assertion rather than dropped, because if the two ever disagree that is worth surfacing: `live_children` treats an unseen pid as *not* recycled (`pid_recycled_into_stranger` returns false for `None`), so on a platform where sysinfo cannot see zombies a pid recycled into a stranger's zombie would be attributed to us and its group killed — the #3067 case. The assertion names that consequence. Not verified on macOS from here, so it is possible the platform gap is real and this converts a timeout into a clear failure at the assertion. That is the intended outcome either way: the failure then points at the mechanism. Refs #3150 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * test(daemon): harden the zombie wait — EINTR, errno, and a bound Three gaps in the `waitid` helper, all in the same direction: a failure that has nothing to do with the child reported as the child failing. `EINTR` is reachable — a signal handler installed without `SA_RESTART` interrupts `waitid` (verified against a C repro; with `SA_RESTART` it is restarted, as signal(7) documents). Nothing in the daemon's lib-test binary installs such a handler today, but that is a property of the current test set, not of the helper. Retry instead. `errno` was discarded, so every failure printed the same line. `ECHILD` (something reaped the wrapper out from under the test) and `EINVAL` (wrong options for this platform) are exactly the answers a macOS triage needs, and this helper exists because of a macOS triage. Dropping the deadline outright traded a 5-second failure with a message for an unbounded block: a wrapper that never exits would hang until the CI job's own timeout killed the run and reported nothing. The wait now runs on its own thread behind a 30-second `recv_timeout`, which bounds the pathological case only — unlike the poll it replaces, it cannot expire while the answer is already available. Also narrows the sysinfo assertion's claim. sysinfo builds a process entry from scratch here (`System::new()`), which on macOS gives up on a zombie; a long-lived `DestroyWait` keeps its `System` across polls and takes the update path instead, which can still see one. The assertion fires on the create path, so its message should not condemn both. Refs #3150 --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
1 parent 7bfda31 commit 8c75fd1

1 file changed

Lines changed: 108 additions & 21 deletions

File tree

binaries/daemon/src/shutdown.rs

Lines changed: 108 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -461,30 +461,117 @@ mod tests {
461461

462462
// Close stdin so `read` hits EOF and the wrapper exits. No `wait()`:
463463
// without the reap it stays a zombie, which is the state under test.
464-
// Spin until the kernel has actually gotten there — callers pause the
465-
// test clock, so this waits on the OS, not on tokio.
466464
drop(wrapper.stdin.take());
467-
let deadline = std::time::Instant::now() + Duration::from_secs(5);
468-
loop {
469-
let mut system = System::new();
470-
system.refresh_processes_specifics(
471-
ProcessesToUpdate::Some(&[Pid::from_u32(pid)]),
472-
true,
473-
ProcessRefreshKind::nothing(),
474-
);
475-
let zombie = system
476-
.process(Pid::from_u32(pid))
477-
.is_some_and(|process| process.status() == ProcessStatus::Zombie);
478-
if zombie {
479-
break;
465+
466+
wait_for_exit_without_reaping(pid);
467+
468+
// The wrapper is now definitively an unreaped zombie. If `sysinfo`
469+
// disagrees, that is a platform gap worth knowing about rather than a
470+
// flake: `live_children` treats an unseen pid as *not* recycled
471+
// (`pid_recycled_into_stranger` returns false for `None`), so a pid
472+
// recycled into a stranger's zombie would be attributed to us and its
473+
// group killed — the #3067 case. Fail loudly and name it.
474+
//
475+
// A failure here is narrower than it first reads, though. This is a
476+
// *fresh* `System`, so sysinfo has to build the entry from nothing,
477+
// and on macOS that path gives up on a process whose argv and exe path
478+
// the kernel no longer serves — which is exactly a zombie. A
479+
// long-lived `DestroyWait` keeps its `System` across polls and has
480+
// already seen the pid alive, so it takes sysinfo's *update* path
481+
// instead, which can still report `Zombie` where the create path
482+
// returns nothing at all. So this bounds how much of the #3067 guard
483+
// survives on a platform; it does not by itself condemn the whole of
484+
// it.
485+
let mut system = System::new();
486+
system.refresh_processes_specifics(
487+
ProcessesToUpdate::Some(&[Pid::from_u32(pid)]),
488+
true,
489+
ProcessRefreshKind::nothing(),
490+
);
491+
let seen = system.process(Pid::from_u32(pid)).map(|p| p.status());
492+
assert_eq!(
493+
seen,
494+
Some(ProcessStatus::Zombie),
495+
"`waitid` confirms the wrapper exited unreaped, but a fresh \
496+
`sysinfo::System` reports {seen:?}. The zombie half of \
497+
`is_live_child` / `pid_recycled_into_stranger` cannot work on \
498+
this platform for a pid sysinfo has not already seen alive."
499+
);
500+
(wrapper, pid)
501+
}
502+
503+
/// Block until the child at `pid` has exited, *without* reaping it.
504+
///
505+
/// `WNOWAIT` leaves the exit status pending, so the process stays an
506+
/// unreaped zombie for the caller's assertions and for the later
507+
/// `Child::wait` in `kill_group_and_reap` to consume.
508+
///
509+
/// This replaces a poll for `ProcessStatus::Zombie` (dora-rs/dora#3150):
510+
/// that made the precondition depend on `sysinfo` *observing* the zombie,
511+
/// which is a platform question, not the one under test. The poll timed
512+
/// out on the macOS nightly while passing on Linux. `waitid` is POSIX and
513+
/// answers the question the helper is actually asking — has it exited yet
514+
/// — with no sleep to lose. Measured on Linux: it returns in about a
515+
/// millisecond.
516+
///
517+
/// The wait runs on its own thread only so that a wrapper which never
518+
/// exits fails *here*, with this message, instead of blocking until the CI
519+
/// job's own timeout kills the run and reports nothing. That deadline
520+
/// bounds the pathological case; it is not part of the normal path, and
521+
/// unlike the poll it replaces it cannot expire while the answer is
522+
/// already available. The thread stays parked in `waitid` if it does fire,
523+
/// which is harmless — the panic is taking the process down anyway.
524+
#[cfg(unix)]
525+
fn wait_for_exit_without_reaping(pid: u32) {
526+
use std::sync::mpsc::RecvTimeoutError;
527+
528+
let (tx, rx) = std::sync::mpsc::channel();
529+
std::thread::spawn(move || {
530+
let result = loop {
531+
// SAFETY: `waitid` only reads the exit state of a child this
532+
// test spawned. Its one out-parameter is `info`, a valid,
533+
// aligned, uniquely borrowed `siginfo_t` that outlives the
534+
// call; zeroing it first is sound because every field is an
535+
// integer, a `pid_t`/`uid_t`, a raw pointer, or padding, for
536+
// all of which all-zeros is a valid bit pattern. `WNOWAIT`
537+
// leaves the status unconsumed, so the child stays reapable.
538+
let rc = unsafe {
539+
let mut info: libc::siginfo_t = std::mem::zeroed();
540+
libc::waitid(
541+
libc::P_PID,
542+
pid as libc::id_t,
543+
&mut info,
544+
libc::WEXITED | libc::WNOWAIT,
545+
)
546+
};
547+
if rc == 0 {
548+
break Ok(());
549+
}
550+
// A signal delivered to this thread must not be reported as
551+
// the child failing to exit — that is the flake class this
552+
// helper exists to remove.
553+
let err = std::io::Error::last_os_error();
554+
if err.kind() == std::io::ErrorKind::Interrupted {
555+
continue;
556+
}
557+
break Err(err);
558+
};
559+
let _ = tx.send(result);
560+
});
561+
562+
match rx.recv_timeout(Duration::from_secs(30)) {
563+
Ok(Ok(())) => {}
564+
// `ECHILD` here would mean something reaped the wrapper out from
565+
// under the test; `EINVAL` would mean the options are wrong for
566+
// this platform. Both are worth telling apart from a timeout.
567+
Ok(Err(err)) => panic!("waitid on the wrapper failed: {err}"),
568+
Err(RecvTimeoutError::Timeout) => {
569+
panic!("the wrapper never exited: waitid blocked for 30s")
570+
}
571+
Err(RecvTimeoutError::Disconnected) => {
572+
panic!("the thread waiting on the wrapper died")
480573
}
481-
assert!(
482-
std::time::Instant::now() < deadline,
483-
"the wrapper never became a zombie"
484-
);
485-
std::thread::sleep(Duration::from_millis(20));
486574
}
487-
(wrapper, pid)
488575
}
489576

490577
#[cfg(unix)]

0 commit comments

Comments
 (0)