Add optional CPU cycle measurement to graviola-bench#210
Conversation
* This was motivated by the observation that the elapsed time for the Graviola benchmarks on macOS can sometimes vary by 10% from one run to the next, probably due to some combination of varying CPU frequency and random background daemons. In past projects, I've found that the CPU's cycle counter, while still noisy, is less noisy than the wall clock. * This change adds a feature `__bench_cycles` that replaces the standard Criterion elapsed time metric with a measurement of elapsed CPU cycles for the process on aarch64. * It should be straightforward to add x86_64 support in a later change, either with inline assembly or by using the Linux `perf` API. (The latter could also provide access to metrics like instructions retired and branch mispredictions.) * Because the counter-timer register is per-core, this implementation may return incorrect results if a single benchmark runs long enough for the OS to reschedule it to a different core or to interrupt the execution with some other process. It may be beneficial to add CPU pinning in the future.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #210 +/- ##
=======================================
Coverage 99.78% 99.78%
=======================================
Files 216 216
Lines 56507 56507
Branches 230 230
=======================================
Hits 56384 56384
Misses 92 92
Partials 31 31 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Merging this PR will degrade performance by 10.38%
|
| Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|
| ❌ | rustcrypto[32B] |
3 µs | 3.4 µs | -10.69% |
| ❌ | rustcrypto[32B] |
3.3 µs | 3.6 µs | -10.06% |
Tip
Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.
Comparing brian-pane:cycles-aarch64 (f4a45d9) with main (9fc602f)
|
I've found an easier way to get repeatable benchmark results on aarch64: on a quiet Linux host, the elapsed times exhibit much less variability than I'm seeing on macOS. |
__bench_cyclesthat replaces the standard Criterion elapsed time metric with a measurement of elapsed CPU cycles for the process on aarch64.perfAPI. (The latter could also provide access to metrics like instructions retired and branch mispredictions.)