Migrate to YAML input format for sending benchmarking data to BenchCloud by AkshayWarrier · Pull Request #1270 · sosy-lab/benchexec

AkshayWarrier · 2026-06-22T15:24:28Z

The PR migrates the format for sending benchmarking data to Benchcloud from the legacy text form to YAML. It also allows users to pass multiple result file patterns, since the new format now supports it.

Must be merged after https://gitlab.com/sosy-lab/software/benchcloud/-/merge_requests/183 is deployed.

PhilippWendler · 2026-06-23T06:29:58Z

Would it be possible to avoid making all paths absolute? I consider this an anti-pattern. Absolute paths may help to avoid path misunderstandings between components, if they are applied everywhere and consistently. But if we have a consistent path handling everywhere, then relative paths with a properly defined base¹ work just as well, so I do not see a real benefit of actual paths. It feels more like "it helped once so now we use absolute paths everywhere". Disadvantages of absolute paths are that eventually they inevitably end up somewhere where they are inconvenient or undesired, for example in logs or result files, and disturb comparisons when debugging, leak private data (user name) on the web, etc. BenchExec for example takes care that it does not introduce absolute paths into output when the user did not use absolute paths as input, and that it never forces the user to use absolute paths as input. I hope BenchCloud does the same.

¹ Standard is that relative paths that are given as command-line arguments or on stdin/stdout are relative to the current working directory, and relative paths given in a file are relative to that file's directory, as long as the involved APIs do not specify something explicitly.

AkshayWarrier · 2026-06-29T12:22:54Z

Hi Philipp, so in BenchCloud, all run specific files, and the toolpaths are relativized against the basedir. And I think to relativize them, they both need to be the same kind of path; maybe that's why they were both made absolute. But maybe we can make all these paths absolute on the BenchCloud client side before relativizing them against the base directory?

An exception is that the output directory is sent as it is to BenchCloud and is not relativized, so maybe we also just make this absolute on the BenchCloud client side then?

PhilippWendler · 2026-06-29T13:50:54Z

But maybe we can make all these paths absolute on the BenchCloud client side before relativizing them against the base directory?

If you require absolute paths there, why not?

An alternative would also be to define for this format that all paths in the file are relative to the given base dir. Though if there is the output directory mixed in, this might not make that much sense.

But maybe it is also a good idea to check whether the grouping of inputs is meaningful? E.g., why is the output directory mixed in with the things that define the runs? IIRC there are some things that are passed to the BenchmarkClient as command-line options and some are passed in this file. How is it decided which is where so far?

AkshayWarrier · 2026-06-29T15:04:16Z

How is it decided which is where so far?

I'm not sure how it has been decided so far, but it might make sense to pass the output-dir as a command-line option, separate from the workload. I guess this would also require me to update the legacy format as such.

An alternative would also be to define for this format that all paths in the file are relative to the given base dir.

Do you mean I enforce benchexec to always send all files relative to the base dir (are they already relative?), and on benchcloud assert all paths are relative to base-dir?

AkshayWarrier · 2026-06-29T15:16:12Z

I guess this would also require me to update the legacy format as such.

Oh, actually, for backward compatibility, the legacy format would accept the output directory from the input file, but starting with the new format, we would have to expect the output directory to be passed by command-line option. Right? 🤔

PhilippWendler · 2026-06-30T05:38:14Z

An alternative would also be to define for this format that all paths in the file are relative to the given base dir.

Do you mean I enforce benchexec to always send all files relative to the base dir (are they already relative?), and on benchcloud assert all paths are relative to base-dir?

Would be a possibility. In BenchExec the paths are not relative to the base dir, relative paths are relative to the current working directory.

Oh, actually, for backward compatibility, the legacy format would accept the output directory from the input file, but starting with the new format, we would have to expect the output directory to be passed by command-line option.

Yes. At least, it is a possibility, not saying you have to do it. I have no overview of which information is passed to the BenchCloud and how/where it is done, so I also do not know what is the best way to pass the output directory. I am just saying it would be good to have a look at all this information and decide where it fits best when one is anyway completely redesigning the input format. Note that in the past there was no clear design of this, information was just put somewhere where it fit.

AkshayWarrier · 2026-07-13T11:34:03Z

Hi Philipp, I've made new changes after our discussion. We decided to pass output-dir through the command-line just for the YAML input format. And all paths passed through the format must be relative, so there's no danger of leaking absolute paths.

The basedir is relative to the current working directory, the logfile is relative to the output-dir, and everything else is relative to basedir.

The code can be reviewed again :)

PhilippWendler

Could you please provide a benchmark definition that exercises all relevant features (runs without input files, with more than one input file, required files specified on all levels of the XML, all limits and requirements, result-file patterns, etc.) and both the resulting cloud input in the old and new format? I think this would greatly help to see that there is nothing forgotten in the new format and that everything is consistent.

PhilippWendler · 2026-07-14T11:03:18Z

+
+    absBaseDir = benchexec.util.common_base_dir(absSourceFiles + absToolpaths)
+    if absBaseDir == "":
+        sys.exit("No common base dir found.")


I think a user would not understand this message, wouldn't they?

When would base dir even be empty?

PhilippWendler · 2026-07-14T11:04:44Z

+        for run in runSet.runs:
+            if os.path.exists(run.identifier):
+                absSourceFiles.extend(map(os.path.abspath, run.sourcefiles))
+            numberOfRuns += 1


I would say the calculation of numberofRuns should be removed from this method. It is not really related to computeBaseDir, and trivial especially using an expression like sum(len(runSet.runs) for ...).

PhilippWendler · 2026-07-14T11:07:35Z

-            break
+    # get limits
+    rlimits = benchmark.rlimits
+    timeLimit = int(rlimits.cputime_hard)


Limits should be ints anyway. But even if we encounter a limit that is not an int, we would not want to silently truncate the value and use a wrong value! So please remove all the int() calls.

PhilippWendler · 2026-07-14T11:10:36Z

+    limits_cpu = {"time": {"hard": timeLimit}}
+    if rlimits.cpu_cores is not None:
+        limits_cpu["cores"] = rlimits.cpu_cores

-            # we assume, that VCloud-client only splits its input at tabs,
-            # so we can use all other chars for the info, that is needed to run the tool.
-            argString = json.dumps(cmdline)
-            assert "\t" not in argString  # cannot call toTabList(), if there is a tab
+    limits = {"cpu": limits_cpu}
+    if rlimits.walltime is not None:
+        limits["walltime"] = int(rlimits.walltime)


Is it really correct that the time limits look like this in the yaml?

limits: cpu: time: hard: 1 walltime: 2

Looks quite inconsistent.

PhilippWendler · 2026-07-14T11:16:40Z

+    if benchmark.result_files_patterns:
+        cloud_input["resultFilePatterns"] = list(benchmark.result_files_patterns)
+
+    cloud_input["runsets"] = runDefinitions


Does the BenchCloud distinguish between different run sets in a single submission?

AFAIU there is just a list of runs per run collection, and all have the same settings such as limits. So it seems redundant to split the input into different run sets and submit the limits for each of them.

Or is the intent to change this? Then we should discuss how to best map the BenchExec structures to what the BenchCloud wants to support.

PhilippWendler · 2026-07-14T11:42:37Z

    return (workingDir, validToolpaths)


+def computeBaseDir(benchmark, absToolpaths):


Wouldn't this need to take all files into account that are part of the input, e.g., also the required files?

PhilippWendler · 2026-07-14T11:45:58Z

+    if benchmark.result_files_patterns:
+        cloud_input["resultFilePatterns"] = list(benchmark.result_files_patterns)


Maybe here it should be explained what the special case for relative patterns is compared to the other relative paths.

And doesn't the BenchCloud forbid absolute patterns? Is this handled somewhere else?

PhilippWendler · 2026-07-14T11:48:52Z

+    cloud_input["memoryreq"] = (
+        r.memory if r.memory is not None else DEFAULT_CLOUD_MEMORY_REQUIREMENT
+    )
+    cloud_input["corereq"] = (
+        r.cpu_cores if r.cpu_cores is not None else DEFAULT_CLOUD_CPUCORE_REQUIREMENT
+    )
+    if r.cpu_model:
+        cloud_input["cpumodels"] = [r.cpu_model]


I find it inconsistent that limits are specified in some nested yaml structure, but requirements are flattened into top-level keys, some of which have the name suffix req and one of which doesn't.

Furthermore, core limits are given as limits: cpu: cores: and core requirements are given as corereq:, which is inconsistent both with respect to plural/singular and with respect to whether "CPU" is an explicit part of the naming.

PhilippWendler · 2026-07-14T11:50:16Z

+        r.cpu_cores if r.cpu_cores is not None else DEFAULT_CLOUD_CPUCORE_REQUIREMENT
+    )
+    if r.cpu_model:
+        cloud_input["cpumodels"] = [r.cpu_model]


Why is cpumodels a list and what are the semantics if there is more than one entry?

Also note that r.cpu_model can already be a list of CPU models (e.g., 'i9-9900, i7-6700'), so we can end up with a list of list of CPU models, which could be confusing and error prone.

PhilippWendler · 2026-07-14T11:51:19Z

-    return "\t".join(map(str, items))
-
-
 def getCloudInput(benchmark):


We need a link here that points to a place where the format for the cloud-input file is described.

Migrate to yaml input format for benchcloud

49a5f4f

AkshayWarrier changed the title ~~Migrate to yaml input format for benchcloud~~ Migrate to YAML input format for sending benchmarking data to BenchCloud Jun 22, 2026

Format source

fa2a7bb

PhilippWendler added the BenchCloud related to the BenchCloud integration label Jun 23, 2026

AkshayWarrier added 3 commits July 6, 2026 17:49

Always send files as relative paths in YAML format

2ef4bfe

Merge branch 'main' into add-yaml-input-format

3a165b0

Remove default timelimit

35efe1b

PhilippWendler reviewed Jul 14, 2026

View reviewed changes

		return (workingDir, validToolpaths)


		def computeBaseDir(benchmark, absToolpaths):

		if benchmark.result_files_patterns:
		cloud_input["resultFilePatterns"] = list(benchmark.result_files_patterns)

		return "\t".join(map(str, items))


		def getCloudInput(benchmark):

Uh oh!

Conversation

AkshayWarrier commented Jun 22, 2026

Uh oh!

PhilippWendler commented Jun 23, 2026

Uh oh!

AkshayWarrier commented Jun 29, 2026

Uh oh!

PhilippWendler commented Jun 29, 2026

Uh oh!

AkshayWarrier commented Jun 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

AkshayWarrier commented Jun 29, 2026

Uh oh!

PhilippWendler commented Jun 30, 2026

Uh oh!

AkshayWarrier commented Jul 13, 2026

Uh oh!

PhilippWendler left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

2 participants

AkshayWarrier commented Jun 29, 2026 •

edited

Loading