Skip to content

Add optional CPU cycle measurement to graviola-bench#210

Closed
brian-pane wants to merge 1 commit into
ctz:mainfrom
brian-pane:cycles-aarch64
Closed

Add optional CPU cycle measurement to graviola-bench#210
brian-pane wants to merge 1 commit into
ctz:mainfrom
brian-pane:cycles-aarch64

Conversation

@brian-pane

Copy link
Copy Markdown
Contributor
  • This was motivated by the observation that the elapsed time for the Graviola benchmarks on macOS can sometimes vary by 10% from one run to the next, probably due to some combination of varying CPU frequency and random background daemons. In past projects, I've found that the CPU's cycle counter, while still noisy, is less noisy than the wall clock.
  • This change adds a feature __bench_cycles that replaces the standard Criterion elapsed time metric with a measurement of elapsed CPU cycles for the process on aarch64.
  • It should be straightforward to add x86_64 support in a later change, either with inline assembly or by using the Linux perf API. (The latter could also provide access to metrics like instructions retired and branch mispredictions.)
  • Because the counter-timer register is per-core, this implementation may return incorrect results if a single benchmark runs long enough for the OS to reschedule it to a different core or to interrupt the execution with some other process. It may be beneficial to add CPU pinning in the future.

* This was motivated by the observation that the elapsed
  time for the Graviola benchmarks on macOS can sometimes
  vary by 10% from one run to the next, probably due to
  some combination of varying CPU frequency and random
  background daemons. In past projects, I've found that
  the CPU's cycle counter, while still noisy, is less
  noisy than the wall clock.
* This change adds a feature `__bench_cycles` that
  replaces the standard Criterion elapsed time metric
  with a measurement of elapsed CPU cycles for the
  process on aarch64.
* It should be straightforward to add x86_64 support
  in a later change, either with inline assembly or
  by using the Linux `perf` API. (The latter could
  also provide access to metrics like instructions
  retired and branch mispredictions.)
* Because the counter-timer register is per-core,
  this implementation may return incorrect results if
  a single benchmark runs long enough for the OS to
  reschedule it to a different core or to interrupt
  the execution with some other process. It may be
  beneficial to add CPU pinning in the future.
@codecov

codecov Bot commented Jul 13, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 99.78%. Comparing base (9fc602f) to head (f4a45d9).

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #210   +/-   ##
=======================================
  Coverage   99.78%   99.78%           
=======================================
  Files         216      216           
  Lines       56507    56507           
  Branches      230      230           
=======================================
  Hits        56384    56384           
  Misses         92       92           
  Partials       31       31           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@codspeed-hq

codspeed-hq Bot commented Jul 13, 2026

Copy link
Copy Markdown

Merging this PR will degrade performance by 10.38%

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

❌ 2 regressed benchmarks
✅ 171 untouched benchmarks

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Benchmark BASE HEAD Efficiency
rustcrypto[32B] 3 µs 3.4 µs -10.69%
rustcrypto[32B] 3.3 µs 3.6 µs -10.06%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.


Comparing brian-pane:cycles-aarch64 (f4a45d9) with main (9fc602f)

Open in CodSpeed

@brian-pane

Copy link
Copy Markdown
Contributor Author

I've found an easier way to get repeatable benchmark results on aarch64: on a quiet Linux host, the elapsed times exhibit much less variability than I'm seeing on macOS.

@brian-pane brian-pane closed this Jul 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant