Skip to content

🧪 test(bench): decide the measurement contracts in peryx-bench-core - #2261

Merged
gaborbernat merged 4 commits into
mainfrom
test/bench-core-mutants-1893
Sep 9, 2026
Merged

🧪 test(bench): decide the measurement contracts in peryx-bench-core#2261
gaborbernat merged 4 commits into
mainfrom
test/bench-core-mutants-1893

Conversation

@gaborbernat

Copy link
Copy Markdown
Member

.cargo/mutants.toml excludes crates/*/src/bench/** on the grounds that a benchmark workload only builds load, so mutating one moves what CodSpeed measures rather than whether anything is correct. peryx-bench-core sits outside that exclusion and earns it. The crate reduces rounds to a median and a coefficient of variation, ranks parties on a logarithmic colour ladder, converts CPU percentages into seconds, and answers the boolean that gates an A/B comparison. The nightly found 75 mutants alive across its six modules; a run against current main put it at 50 alive, 8 of them cfg twins this host cannot build, plus 4 that finished with no verdict.

None of the six modules only shapes load. stats.rs, report.rs and compare.rs reduce, format and classify, with no I/O. usage.rs reports resident memory and CPU seconds for a process tree, and servers.rs decides when a spawned server is ready. machine.rs is the closest call, since four of its functions exist to build load, but each returns a rate that lands in the published machine.toml, so the arithmetic behind that rate carries a contract like any other. This PR adds no exclusion and proposes none.

Four survivors resist any test as written, so the code changes instead. bytes / workers and each * workers appeared twice each in machine.rs and moved how much load a measurement built while leaving the rate it reported alone; folding them into shares gives the round-down a value a test can name. write_one drove its loop off a running total, so a mutated increment ran forever and no timeout could book it; walking spans off the offsets bounds each write by construction. tree_of walked up from every process and needed a self-parent comparison to terminate, which no test can construct; walking down from the root and taking each parent's children once terminates by construction and visits each process once instead of once per descendant. wait_ready bounded its own poll loop with an Instant comparison that tokio::time::timeout says with no arithmetic to get wrong.

Three more needed the code to hand back what it had already worked out. drain reports the bytes it read, so the loopback rate comes from what arrived. The CPU-percent conversion moved into cpu_millis and the printed change column into describe, since a value that reaches stdout inside a format string and nowhere else is a value nothing can read back. kill_process_group returns the io::Result it was discarding, and Drop goes on ignoring it.

Two more belonged to the harness. A server that never became ready left an accept pending in the fixture thread, and the guard joined that thread forever, so three mutants that should have failed a test hung it instead. The guard now releases the accepts it did not use.

That leaves eight rows in a local missed.txt, all of them the not(target_os = "macos") halves of model, model_at, cores and describe_cores. Rewriting those gates to cfg(all()) and building the host that way settles all eight: the existing suite catches model_at and describe_cores, and model and cores are unviable, because replacing either body orphans its only caller and dead_code fails the build. Nothing else in the crate survives on hardware that can compile it.

Refs #1893

Every number machine.rs writes into the published profile — the host row, the
mount it measured, the four baseline rates — was reachable but undecided. A run
of cargo-mutants over the file left 58 mutants alive, including ones that
report a rate of zero, read a different sysctl, or write a file of the wrong
length.

Four of the survivors could not be decided as written, so the code changes
rather than the tests. `bytes / workers` and `each * workers` appeared twice
each and only ever moved how much load a measurement builds, never the rate it
reports; folding them into `shares` makes the round-down a value a test can
name. `write_one` drove its loop off a running total, so a mutated increment
never terminated and no timeout could book a verdict; walking `spans` off the
offsets bounds every write by construction. `loopback_http` multiplied the
payload by the client count instead of adding up what the streams actually
read, and `mount_for` inlined the exact-match lookup that decides whether df's
answer is used at all.

Refs #1893
The regression gate, the cell tints, the resource sampler and the readiness
wait all had reachable code no assertion could tell apart. A cargo-mutants run
over the crate left the log ladder undetermined, the CPU conversion unchecked,
the 3% threshold decided only from the failing side, and four mutants with no
verdict at all because the suite hung instead of failing.

The hangs were the fixture, not the product: a server that never became ready
left an accept pending and the guard joined that thread forever. The guard now
releases the accepts it did not use. `wait_ready` bounded its own poll loop
with an `Instant` comparison that a mutation turned into an immediate bail;
`tokio::time::timeout` says the same thing with no arithmetic to get wrong.

Three more survivors needed the code to expose what it already computed.
`tree_of` walked up from every process and needed a self-parent test to
terminate, which no test can construct; walking down from the root and taking
each parent's children once terminates by construction and visits each process
once. The CPU-percent conversion and the printed change column both moved to
named functions, because a value that only reaches stdout inside a format
string is a value no test can read.

Refs #1893
Port 0 asks the kernel for any port, which is not an answer a caller building
a base URL can use. Nothing said so, so replacing the lookup with that
sentinel left every server test polling a dead address until the three-minute
startup budget ran out, and the suite reported a timeout instead of a failure.

Refs #1893
The process-tree sample ran against a test process that had no children, so
the descendant walk never executed a line. Spawning a child that blocks on
stdin gives the sample a real tree and pins that a descendant is counted while
an unrelated process is not.

Refs #1893
@gaborbernat gaborbernat added type:test Testing, conformance, metrics, or validation coverage area:tests Conformance, perf tests, and observability priority:P2 Performance, observability, or deferred feature labels Sep 9, 2026
@codspeed-hq

codspeed-hq Bot commented Sep 9, 2026

Copy link
Copy Markdown

Merging this PR will not alter performance

✅ 29 untouched benchmarks
⏩ 133 skipped benchmarks1


Comparing test/bench-core-mutants-1893 (1967611) with main (34eca99)

Open in CodSpeed

Footnotes

  1. 133 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports.

@gaborbernat
gaborbernat merged commit ff8ee95 into main Sep 9, 2026
22 checks passed
@gaborbernat
gaborbernat deleted the test/bench-core-mutants-1893 branch September 9, 2026 18:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:tests Conformance, perf tests, and observability priority:P2 Performance, observability, or deferred feature type:test Testing, conformance, metrics, or validation coverage

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant