Leaky verdict on benchmark_cli test can fail the ci profile

[gaborbernat]

Problem

bench_cli benchmark_cli_exposes_oci_options is reported leaky by nextest under a full instrumented run, and the ci profile turns that into a failure:

leak-timeout = { period = "1s", result = "fail" }

In isolation under --profile ci the same test passes. So the leak is contention-dependent: it appears when the test runs alongside others and not when it runs alone. That makes it a non-deterministic failure of just coverage-native, which is the job every PR's coverage check runs.

Observed during a full-crate instrumented pass while investigating #2181. That run used the default profile, where leaky is a warning rather than a failure, which is why it surfaced as an observation rather than a red job.

Why it is filed separately

It is a different test and a different mechanism from #2181, but the same class: a latent fragility that only appears under contention and that the ci profile promotes from warning to failure. #2181 is a timeout; this is a leak verdict.

It is also distinct from #2160. There the leak lands on tests that hold no descriptor they could leak, which points at a capture pipe inherited from a concurrent spawn — the upstream race in nextest#3553. This test drives a CLI, so whether it genuinely holds a descriptor open past the test body needs establishing rather than assuming.

Required change

Establish first whether the test leaks a real handle or inherits one.

If it holds something open — a child process not reaped, a file or socket outliving the assertion — close it, and the fix is ordinary. If it inherits a capture pipe from a concurrent spawn, this is nextest#3553 and belongs with #2160 rather than here; say so and close this as a duplicate with the evidence.

Do not raise leak-timeout, and do not move the test out of the ci profile's reach. Both make the report go away without answering which of the two it is.

Acceptance criteria

• The test passes under --profile ci in a full parallel run, not only in isolation.
• The verdict rests on evidence about what the test holds open, not on a timing adjustment.
• leak-timeout keeps result = "fail" in the ci profile.

Boundary

This test and whatever it holds. Do not change the leak policy, and do not touch #2181's bound or #2219's port acquisition.

[gaborbernat]

Not reproducible, and the priority argument does not hold

Three instrumented runs of the whole crate, one leak, and it was the first:

run	branch	tests	profile	leak window	result
original observation	test/mutants-oci-1893	1314	default	100ms	LEAK at 1313/1314
reproduction	fix/bench-port-handover-2219	1296	ci	1s	clean
controlled	fix/bench-port-handover-2219	1296	default	100ms	clean

The third repeats the first's conditions — instrumented, whole crate, default profile, same 100ms window — and does not leak. So the verdict is not a property of instrumentation, and not a property of the profile either. It was a single timing event at position 1313 of 1314, at the tail of a busy run where stragglers pile up.

Two hypotheses died here. The first, that the test inherits a capture pipe from a concurrent spawn, would make this a duplicate of #2160 and nextest#3553. The second, mine, was that ci's 1s window versus the default 100ms was the single variable. The controlled run rules that out: same window, same build, no leak.

The premise this issue was filed on is backwards. It says the leak can fail the coverage job non-deterministically. just coverage-native runs --profile ci, and .config/nextest.toml sets leak-timeout = { period = "1s", result = "fail" } there, while the default profile inherits nextest's 100ms. The coverage job's window is ten times wider than the one the leak appeared under, so of the two configurations it is the less exposed. The reasoning took result = "fail" and stopped before the period.

Not a duplicate of #2160. Whatever this was, the mechanism is not an inherited pipe. This test spawns peryx-bench-oci, a workspace Rust binary, so under cargo llvm-cov the child is itself instrumented, inherits LLVM_PROFILE_FILE and writes a .profraw from an atexit handler. .output() returns at pipe EOF rather than at reap, which leaves a plausible gap in which an instrumented child is still finishing exit work. That remains a hypothesis: it predicts a leak under instrumentation, and two instrumented runs since have not produced one.

The part worth keeping: this shape is not one test

cargo_binary has callers in 14 files. Eight call .output() on the child:

peryx/tests/cli_entrypoint.rs           peryx-test-support/tests/fixture.rs
peryx/tests/process_fixture.rs          peryx-test-support/tests/unit/tests.rs
peryx/tests/policy_reference.rs         peryx-ecosystem-pypi/tests/system-suite/tests/cases/s3_upload.rs
peryx-ecosystem-oci/tests/bench_cli.rs  peryx-ecosystem-pypi/tests/bench_cli.rs

benchmark_cli_exposes_pypi_suite is a near-identical twin: same --help, same .output(), spawning peryx-bench-pypi. Anything true of the oci test is true of it.

If a fix is ever wanted, the seam is a shared spawn-and-reap helper beside cargo_binary in peryx-test-support, not eight separate corrections. It is not wanted on this evidence: a child that finishes its exit work inside the gate's own window is not broken, and .output() returning at EOF rather than at reap is ordinary std behaviour.

For anyone re-measuring

#2226 merged after these runs and throttles the six bench server tests to one thread, which changes the contention profile of exactly this kind of measurement. A clean run on current main is therefore weaker evidence than these were, and should not be read as a refutation of a future observation.

Suggest closing this as not reproducible, with the class analysis above kept for whoever next sees a leak verdict on one of those eight.