🧪 test(bench): decide the measurement contracts in peryx-bench-core - #2261
Merged
Conversation
Every number machine.rs writes into the published profile — the host row, the mount it measured, the four baseline rates — was reachable but undecided. A run of cargo-mutants over the file left 58 mutants alive, including ones that report a rate of zero, read a different sysctl, or write a file of the wrong length. Four of the survivors could not be decided as written, so the code changes rather than the tests. `bytes / workers` and `each * workers` appeared twice each and only ever moved how much load a measurement builds, never the rate it reports; folding them into `shares` makes the round-down a value a test can name. `write_one` drove its loop off a running total, so a mutated increment never terminated and no timeout could book a verdict; walking `spans` off the offsets bounds every write by construction. `loopback_http` multiplied the payload by the client count instead of adding up what the streams actually read, and `mount_for` inlined the exact-match lookup that decides whether df's answer is used at all. Refs #1893
The regression gate, the cell tints, the resource sampler and the readiness wait all had reachable code no assertion could tell apart. A cargo-mutants run over the crate left the log ladder undetermined, the CPU conversion unchecked, the 3% threshold decided only from the failing side, and four mutants with no verdict at all because the suite hung instead of failing. The hangs were the fixture, not the product: a server that never became ready left an accept pending and the guard joined that thread forever. The guard now releases the accepts it did not use. `wait_ready` bounded its own poll loop with an `Instant` comparison that a mutation turned into an immediate bail; `tokio::time::timeout` says the same thing with no arithmetic to get wrong. Three more survivors needed the code to expose what it already computed. `tree_of` walked up from every process and needed a self-parent test to terminate, which no test can construct; walking down from the root and taking each parent's children once terminates by construction and visits each process once. The CPU-percent conversion and the printed change column both moved to named functions, because a value that only reaches stdout inside a format string is a value no test can read. Refs #1893
Port 0 asks the kernel for any port, which is not an answer a caller building a base URL can use. Nothing said so, so replacing the lookup with that sentinel left every server test polling a dead address until the three-minute startup budget ran out, and the suite reported a timeout instead of a failure. Refs #1893
The process-tree sample ran against a test process that had no children, so the descendant walk never executed a line. Spawning a child that blocks on stdin gives the sample a real tree and pins that a descendant is counted while an unrelated process is not. Refs #1893
Merging this PR will not alter performance
Comparing Footnotes
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
.cargo/mutants.tomlexcludescrates/*/src/bench/**on the grounds that a benchmark workload only builds load, so mutating one moves what CodSpeed measures rather than whether anything is correct.peryx-bench-coresits outside that exclusion and earns it. The crate reduces rounds to a median and a coefficient of variation, ranks parties on a logarithmic colour ladder, converts CPU percentages into seconds, and answers the boolean that gates an A/B comparison. The nightly found 75 mutants alive across its six modules; a run against currentmainput it at 50 alive, 8 of themcfgtwins this host cannot build, plus 4 that finished with no verdict.None of the six modules only shapes load.
stats.rs,report.rsandcompare.rsreduce, format and classify, with no I/O.usage.rsreports resident memory and CPU seconds for a process tree, andservers.rsdecides when a spawned server is ready.machine.rsis the closest call, since four of its functions exist to build load, but each returns a rate that lands in the publishedmachine.toml, so the arithmetic behind that rate carries a contract like any other. This PR adds no exclusion and proposes none.Four survivors resist any test as written, so the code changes instead.
bytes / workersandeach * workersappeared twice each inmachine.rsand moved how much load a measurement built while leaving the rate it reported alone; folding them intosharesgives the round-down a value a test can name.write_onedrove its loop off a running total, so a mutated increment ran forever and no timeout could book it; walkingspansoff the offsets bounds each write by construction.tree_ofwalked up from every process and needed a self-parent comparison to terminate, which no test can construct; walking down from the root and taking each parent's children once terminates by construction and visits each process once instead of once per descendant.wait_readybounded its own poll loop with anInstantcomparison thattokio::time::timeoutsays with no arithmetic to get wrong.Three more needed the code to hand back what it had already worked out.
drainreports the bytes it read, so the loopback rate comes from what arrived. The CPU-percent conversion moved intocpu_millisand the printed change column intodescribe, since a value that reaches stdout inside a format string and nowhere else is a value nothing can read back.kill_process_groupreturns theio::Resultit was discarding, andDropgoes on ignoring it.Two more belonged to the harness. A server that never became ready left an accept pending in the fixture thread, and the guard joined that thread forever, so three mutants that should have failed a test hung it instead. The guard now releases the accepts it did not use.
That leaves eight rows in a local
missed.txt, all of them thenot(target_os = "macos")halves ofmodel,model_at,coresanddescribe_cores. Rewriting those gates tocfg(all())and building the host that way settles all eight: the existing suite catchesmodel_atanddescribe_cores, andmodelandcoresare unviable, because replacing either body orphans its only caller anddead_codefails the build. Nothing else in the crate survives on hardware that can compile it.Refs #1893