Benchmark script to wrap dtest

[tameware]

Sample output:

$ ./benchmark.sh 
DDS dtest benchmark
===================
branch:      /Users/adamw/src/dds/bazel-bin/library/tests/dtest
hands dir:   /Users/adamw/src/dds/hands
max_deals:   100
files:       list100.txt list10.txt list1.txt
git branch:  benchmark
repeats:     1

solver file              ver  user_ms   sys_ms   avg_user  ratio run
------ ------------- ------- -------- -------- ---------- ------ ---
solve  list100.txt    branch      284     2428       2.84   8.55 1/1
solve  list10.txt     branch       47      133       4.70   2.83 1/1
solve  list1.txt      branch       15       15      15.00   1.00 1/1
calc   list100.txt    branch     1457    14900      14.57  10.23 1/1
calc   list10.txt     branch      237     1178      23.70   4.97 1/1
calc   list1.txt      branch       45      149      45.00   3.31 1/1

Completed 6 runs (6 expected).

Compare against a binary from another branch or v2.9. Comparing against a branch could be automated; as is I build a dtest in another branch, rename it, and move it to dds/..

$ ./benchmark.sh --build --compare ../dtest_solver_context_reuse --max-deals 100
Building //library/tests:dtest...
Starting local Bazel server (9.1.0 Homebrew) and connecting to it...
INFO: Analyzed target //library/tests:dtest (126 packages loaded, 1424 targets configured).
INFO: Found 1 target...
Target //library/tests:dtest up-to-date:
  bazel-bin/library/tests/dtest
INFO: Elapsed time: 3.827s, Critical Path: 0.02s
INFO: 1 process: 96 action cache hit, 1 internal.
INFO: Build completed successfully, 1 total action
DDS dtest benchmark
===================
branch:      /Users/adamw/src/dds/bazel-bin/library/tests/dtest
compare:     ../dtest_solver_context_reuse
run order:   branch, compare
hands dir:   /Users/adamw/src/dds/hands
max_deals:   100
files:       list100.txt list10.txt list1.txt
git branch:  benchmark
repeats:     1

solver file              ver  user_ms   sys_ms   avg_user  ratio run
------ ------------- ------- -------- -------- ---------- ------ ---
solve  list100.txt    branch      300     2417       3.00   8.06 1/1
solve  list100.txt   compare      246     2450       2.46   9.96 1/1
solve  list10.txt     branch       47      127       4.70   2.70 1/1
solve  list10.txt    compare       50      134       5.00   2.68 1/1
solve  list1.txt      branch       15       15      15.00   1.00 1/1
solve  list1.txt     compare       16       16      16.00   1.00 1/1
calc   list100.txt    branch     1443    14842      14.43  10.29 1/1
calc   list100.txt   compare     1408    14731      14.08  10.46 1/1
calc   list10.txt     branch      240     1123      24.00   4.68 1/1
calc   list10.txt    compare      241     1173      24.10   4.87 1/1
calc   list1.txt      branch       46      147      46.00   3.20 1/1
calc   list1.txt     compare       45      147      45.00   3.27 1/1

Summary (branch vs compare, avg user ms; cmp/branch > 1 => branch faster)
==============================================================================
solver file           compare_avg   branch_avg cmp/branch note           
------ ------------- ------------ ------------ ---------- ---------------
solve  list100.txt           2.46         3.00      0.82x compare faster 
solve  list10.txt            5.00         4.70      1.06x branch faster  
solve  list1.txt            16.00        15.00      1.07x branch faster  
calc   list100.txt          14.08        14.43      0.98x compare faster 
calc   list10.txt           24.10        24.00      1.00x branch faster  
calc   list1.txt            45.00        46.00      0.98x compare faster 

Completed 12 runs (12 expected).

[Copilot]

Pull request overview

Adds a new benchmark.sh helper script to benchmark library/tests/dtest across solver modes and hand-file sizes, with optional building and binary-to-binary comparison.

Changes:

• Introduces a Bash benchmark runner that iterates solver/file combinations and prints per-run timing rows.
• Adds optional --build, --compare, --repeats, and --max-deals controls plus -- pass-through args to dtest.
• Adds an aggregated comparison summary when a second binary is provided.

[Copilot]

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 1 comment.

[Copilot]

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 1 comment.

[Copilot]

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 1 comment.

[zzcgumn]

Interesting idea. I am not sure about adding this to the CI run as we don't yet have an idea about how large performance regressions we can tolerate.

I did a couple of comparisons like this manually while refactoring from 2.9 to 3.0. The main problem I found was methods that had been inlined but were not in my release candidate code.

[tameware]

I had not considered adding this to the CI run. I like the idea! Something simpler could work for CI, wrap dtest and fail if the runtime for any of list1/10/100 or calc 1/10/100 becomes slower by more than a threshold. There's a fair bit of random variation from run to run, though, and some non-random order variation. On my Mac, the current version speeds up by more than 10% when the machine is "warm", in particular if running solve after many calc runs.

[Copilot]

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 2 comments.

[Copilot]

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 1 comment.