The benchmark generally contains large amount of experiments. Even if the tasks are fairly short the execution of the benchmark overall takes a very long time unless we parallelize it. It is possible to execute the same benchmark multiple times in parallel but it is cumbersome. We should support parallelizing the execution of the experiments while executing benchmark.process, by reusing an Executor object.
The trickiest part is that some assessments (ex ParallelAssessment) can also parallelize the execution of trials. These assessments' executor should not be starved by running too many experiments in parallel, thus we need to account for the ressources needed by the assessment when dispatching the experiment.
The benchmark generally contains large amount of experiments. Even if the tasks are fairly short the execution of the benchmark overall takes a very long time unless we parallelize it. It is possible to execute the same benchmark multiple times in parallel but it is cumbersome. We should support parallelizing the execution of the experiments while executing benchmark.process, by reusing an Executor object.
The trickiest part is that some assessments (ex ParallelAssessment) can also parallelize the execution of trials. These assessments' executor should not be starved by running too many experiments in parallel, thus we need to account for the ressources needed by the assessment when dispatching the experiment.