Release 3.35

We are planning to remove support for cgroups v1 from BenchExec in April 2027.
More information is in issue 1267, please respond there if this is problematic for you.

• When specifying hyperlinks in table definitions for table-generator, one can now use ${value} in addition to the existing variables in order to refer to the cell's value.
• Fix bug where directory modes for symlinks were silently ignored, now BenchExec produces an error if this is attempted.
• Several new and improved tool-info modules, in particular for SAT solvers.
• Most of the JavaScript code for our HTML tables was migrated to TypeScript.
  There should be no user-visible changes except for a few small bug fixes, but we are thankful to @Simon1375 for this valuable maintenance improvement!

Release 3.34

• One new tool-info module.

Release 3.33

• One new tool-info module.

Release 3.32

• Updated tool-info modules for SV-COMP'26 and Test-Comp'26.

Release 3.31

BenchExec now requires Python 3.10 or newer.

• Provide command-line arguments for choosing result files of benchexec.
  So far benchexec writes result files for every <rundefinition> and <tasks> tag. This can be convenient but also redundant, and there is an inconsistency in case of a run definition with a single task set. To solve this, we introduce two new command-line arguments:

  • --results-per-rundefinition
  • --results-per-taskset

  Using them allows one to choose the desired result files, either only one set of files or both. If none of the arguments is given the previous default behavior is used, though in BenchExec 4.0 we will change the default behavior to a more consistent one.

• Duplicate names of <rundefinition> tags in the benchmark definition are now forbidden.
  It was an oversight that this was allowed, because it would lead to output files with duplicate names, i.e., some results overwriting other results. Now benchexec detects and prevents this. Until the next major release duplicate run-definition names are still allowed in cases where it does not cause actual problems (e.g., because only one of them is selected for execution). In BenchExec 4.0 we will likely forbid such duplicate names completely.

• Prevent overwriting results when <tasks> tags with duplicate names exist in the benchmark definition.
  Such cases lead to output files with duplicate names, i.e. some results overwriting other results. Now benchexec detects this, warns about it, and skips writing the files with duplicate names. In BenchExec 4.0 we will likely forbid such duplicate names completely.

• Fix crash on VMs where the CPU frequency is unknown.
  It seems this happens on some VMs on ARM Macs. Thanks to @leventeBajczi for reporting and fixing this!

• Fix crash in special cases of cgroupsv2 systems with a memory limit defined outside of BenchExec (e.g., by the system administrator).

• Fix crash in case of running BenchExec inside a container with an incomplete cgroupsv2 setup where we are in an out-of-tree cgroup.

• Some performance improvements for large benchmark definitions that include task definitions more than once.

• Many new and improved tool-info modules.

Release 3.30

This will be the last release of BenchExec to support Python 3.8 and 3.9.
Future versions will require Python 3.10.

• Rename statistics row "local summary" in HTML tables to "summary measurements" and improve tooltip explanation to make it better understandable.
  Also add a command-line argument --no-local-summary to table-generator to be able to hide it completely.
• Fix crash on cgroupv2 systems where some cgroupsv1 controllers are not present in the kernel anymore (e.g., Fedora 41 or newer).
  Thanks to our GSoC participant @t0hsumi for investigating and fixing this!
• Fix visibility of bind mounts in container mode with fuse-overlayfs.
• Improved explanation in error message about forbidden user namespaces.
• Some new and improved tool-info modules.

Note: We now recommend all users of BenchExec on modern systems (kernel 5.19 or newer) to use cgroupsv2 instead of the outdated cgroupsv1.

Release 3.29

• The name of the wheel and tar archives in BenchExec releases changes from BenchExec-$VERSION* to benchexec-$VERSION* in accordance with Python standard PEP 625 and PyPI requirements.
• Some new and improved tool-info modules.

Release 3.28

• Some new and improved tool-info modules.

Release 3.27

• Support for version 2.1 of the task-definition format.
• Updated tool-info modules for SV-COMP'25 and Test-Comp'25.

Release 3.26

This release brings several important fixes for cgroups v2 and all users on systems with cgroups v2 are strongly recommended to upgrade.

• Fix regression from BenchExec 3.25 for detecting memory-limit violations of a run.
  The changes related to the LXCFS integration in the last version caused the problem that a run that hit the memory limit was no longer marked as "out of memory" but as "failed" and did not have any measurement results (only for cgroups v2 and with container mode enabled).
  Note that the memory limit was still correctly working and also measurements produced for non-failed runs were correct.
• Fix AssertionError when creating a systemd scope for cgroups v2 usage.
  This was caused by a race condition and did occur only with a certain probability depending on the system load.
• Fix warning about left-over cgroup during run cleanup for cgroups v2.
• More robust cleanup of cgroups created by the benchmarked process.
• Fix installation of our Python package on Windows.
• Several new and improved tool-info modules.