Bug: rare missed wakeup under high concurrency (only in Azure?)

[lomotos10]

Description of the issue

Description:

cargo nextest list (and cargo nextest run, which performs the same test discovery phase) hangs indefinitely when discovering tests across a large workspace (~190 test binaries). Forensic thread/process dumps taken at the moment of the hang consistently show:

• One specific test binary's child process is blocked in the kernel with wchan=pipe_write, i.e. it filled its stdout pipe (which nextest opens as Stdio::piped()) and is waiting for the reader to drain it.
• The cargo-nextest parent process has all of its tokio worker threads parked in futex_wait_queue_me (i.e. idle), including at a point where my sampler shows only one discovery child (list_children=1) is still outstanding — so the parent is not "busy," it is simply never resuming the read on that one remaining child's stdout.
• This produces a genuine circular wait: nextest list cannot return until every binary's discovery future completes; the stuck binary cannot exit until its stdout is drained; nothing is draining it.

This binary is always the one with the largest --list --format terse output in my workspace (several thousand tests, several hundred KB of text) — i.e. the only binary whose output can actually exceed the OS pipe buffer (64 KiB default on Linux) and therefore actually block on write(). Binaries with smaller output never trigger this, since they can write their entire listing into the buffer without blocking, regardless of whether the reader is prompt.

I was able to reproduce this reliably in my real CI environment with production-equivalent resources, but have not been able to build a minimal, self-contained reproduction outside that environment — see Additional context.

Steps to reproduce:

1. Have a Cargo workspace large enough that cargo nextest list discovers many binaries concurrently (~190 in my case — concurrency is get_num_cpus(), 28 in my case), where at least one binary's --list --format terse output exceeds the pipe buffer (in my case, one binary has ~5,000 tests and several hundred KB of listing output).
2. Extract a cargo nextest archive once, then repeatedly invoke cargo nextest list against it back-to-back (I used --binaries-metadata/--cargo-metadata/--workspace-remap/--target-dir-remap to reuse the extracted archive and skip re-extraction; a normal invocation takes ~2 seconds).
3. After some number of iterations, one invocation hangs indefinitely instead of returning. In my case, individual hangs occurred anywhere between iteration ~125 and ~320 of 1,000-iteration runs; aggregated across 50 independent runs (50,000 attempts total), the per-invocation hang rate was roughly 0.04%.

Expected outcome

cargo nextest list returns the discovered test list within a couple of seconds, as it does on every other invocation.

Actual result

The invocation hangs indefinitely (I've only ever seen it recover via an external timeout/kill — nothing progresses on its own). Representative forensic excerpt from one hung iteration, captured by a watchdog that dumps /proc/<pid>/status, /proc/<pid>/wchan, ps process trees, and a 5ms-interval process-state sampler when an iteration exceeds 300s:

timeout after 300s: nextest_iteration=159 pid=40697

# process tree at time of hang:
    PID    PPID STAT ELAPSED WCHAN                            COMMAND
  40754   40697 S        302 pipe_write                       .../deps/main-0e3366e0ff26b063 --list --format terse

# /proc/40754/wchan:
pipe_write

# /proc/40754/fd:
lr-x------ 1 root root 64 ... 0 -> /dev/null
l-wx------ 1 root root 64 ... 1 -> pipe:[666972]
l-wx------ 1 root root 64 ... 2 -> pipe:[666973]

# parent (cargo-nextest, pid 40697) thread states, all 29 tokio worker threads:
State: S (sleeping)
wchan: futex_wait_queue_me
[... repeated for every worker thread; zero threads have wchan=pipe_read or similar ...]

# 5ms-interval sampler heartbeat in the seconds leading up to the timeout
# (steady state — nothing changes for 40+ seconds before the watchdog fires):
+265.159s heartbeat parent_state=S (sleeping) parent_threads=29 parent_rss=42288 kB list_children=1
+270.162s heartbeat parent_state=S (sleeping) parent_threads=29 parent_rss=42288 kB list_children=1
+275.166s heartbeat parent_state=S (sleeping) parent_threads=29 parent_rss=42288 kB list_children=1
...
+305.185s heartbeat parent_state=S (sleeping) parent_threads=29 parent_rss=42288 kB list_children=1

The list_children=1 in the heartbeat is the important detail: by the time the hang is frozen, all ~189 other discovery children have already completed. The parent isn't overloaded or busy — it has exactly one outstanding child, and it is simply not polling/reading it anymore. Worth being precise about what "not draining it" means here: the pipe's kernel buffer is full of unread bytes, so a read() on that fd would return data immediately if attempted — nothing is preventing the parent from making progress at the OS level. It just never gets around to trying.

Every single one of the ~16+ reproduced hangs, and the original production hangs that led me to investigate this, show the exact same signature: the largest-output binary stuck in pipe_write, parent tokio runtime fully idle.

Nextest version

cargo-nextest 0.9.137 (75ddba7e9 2026-05-26)
release: 0.9.137
commit-hash: 75ddba7e911b44c5c0700dac0415d824403de9bd
commit-date: 2026-05-26
host: x86_64-unknown-linux-gnu

Additional context

Environment:

• nextest-runner 0.118.0; tokio 1.52.3, mio 1.2.0 (as pinned in the cargo-nextest-0.9.137 release Cargo.lock)
• Linux x86_64, Ubuntu 22.04 (jammy) container image, running on Azure Kubernetes Service (AKS)
• cgroup CPU quota: 28 cores, memory limit 110 GiB

Suspected code path:

Looking at nextest-runner/src/list/test_list.rs on cargo-nextest-0.9.137:

let stream = futures::stream::iter(test_artifacts).map(|test_binary| async {
    // ...
    let (non_ignored, ignored) = test_binary.exec(&lctx, &list_settings, ctx.target_runner).await?;
    // ...
});
let fut = stream.buffer_unordered(list_threads).try_collect::<Vec<_>>();
let parsed_binaries = runtime.block_on(fut)?; // runtime = tokio::runtime::Runtime::new() (multi-thread)

exec (also in test_list.rs) is where the two children per binary come from:

async fn exec(&self, ...) -> Result<(String, String), CreateTestListError> {
    // ...
    let non_ignored = self.exec_single(false, lctx, list_settings, platform_runner);
    let ignored = self.exec_single(true, lctx, list_settings, platform_runner);
    let (non_ignored_out, ignored_out) = futures::future::join(non_ignored, ignored).await;
    Ok((non_ignored_out?, ignored_out?))
}

Each exec_single call (--list --format terse and --list --format terse --ignored) goes through TestCommand::wait_with_output (nextest-runner/src/test_command.rs):

cmd.stdout(std::process::Stdio::piped()).stderr(std::process::Stdio::piped());
let res = tokio::process::Command::from(cmd).spawn();
res?.wait_with_output().await

So at any given moment, up to list_threads * 2 child processes are alive, each with 2 pipes registered with the tokio/mio reactor, and this churns rapidly as ~190 binaries flow through the buffer_unordered window. list_threads is hardcoded to get_num_cpus() at the call site in cargo-nextest/src/dispatch/core/filter.rs, which resolves to my cgroup's CPU quota (28), so up to 56 concurrent children in my case.

My working hypothesis is that under this level of concurrent process spawn/reap/pipe-registration churn, the readiness registration/wakeup for one specific pipe fd (the stdout of the largest-output binary, which is the only one that actually blocks on write() because its output exceeds the pipe buffer) is lost or never re-armed, so the corresponding AsyncRead future for that child's stdout is never polled again. I don't have a confirmed root cause inside tokio/mio itself — the forensic evidence tells me where and how it manifests, but not why the specific reactor wakeup is lost.

Notes:

I also tried to build a minimal reproduction outside my CI environment — small dummy binaries, one of them printing enough terse test names to exceed the pipe buffer, alongside ~150-200 other dummy binaries, run through cargo nextest list in a loop — but wasn't able to trigger the hang locally after a reasonable number of iterations. I don't know why; my synthetic setup differs from my real workspace in more ways than I've been able to isolate (real dependency graph, real binary sizes, real distribution of per-binary output sizes, etc.), and I haven't pinned down which of those differences (if any) actually matters.

Flagging this in case the signature — a large-output binary stuck in pipe_write while cargo-nextest's own tokio runtime sits fully idle — matches something already known, or is worth a closer look. I have more forensic data from the CI reproductions (additional hung runs, per-thread dumps) if it'd help.

[sunshowers]

Could you please write this in your own words without AI slop bloat? Thanks.

Also it's very hard to debug a lost wakeup without some kind of repro. Given how rarely this occurs it is more likely a bug in one of the underlying systems (possibly even the kernel) than in nextest. I'm not sure how to proceed from here.

Have you been able to reproduce this locally at all? On a different kernel version? I think the first thing to do would be to try and use elimination to figure out whether the kernel version is at fault, the cgroup, and so on.

(It is fine to use AI to help you do the analysis, but the final analysis should be in your own words. If English isn't a first language for you, that's fine — I would much rather read concise English that isn't quite perfect than a big wall of AI text that doesn't get to the point. If that is uncomfortable, please use your native language and I'll use an LLM on my end to translate.)

[sunshowers]

I spent some time trying to repro this under various scenarios (including deliberately adding delays in tokio and mio) and couldn't get a repro. I think the next step is to isolate the bug:

• local vs your cloud environment (vs other cloud environments if possible)
• bare metal vs container
• different kernel versions
• writing a binary spawner of your own