Investigate number of samples required of simulator

[henrikstoklandberg]

Business need

• when we create an experiment using our helpers we specific n_simulations_per_point which controls how many samples are taken of the simulator to produce the distribution estimate. This number is currently very high as it removes a source of uncertainty - but this is unrealistic.
• Need to set this number lower and investigate the impact on speed of DOE convergence, Final uncertainty it becomes possible to reduce to etc.

 rax.experiment.make_experiment(sim, SEARCH_SPACE, DIST, n_simulations_per_point=10_000)

Definition of done:

• QOI and DOE results with lower value for parameter. Ideal a value around 20~30, want to find the lowest value where we can still get reasonable results.
• Write up for users of what they should consider when setting this parameter. How should they consider the trade off between decrease runtime per point (when n_simulations_per_point is lower) vs. (??? the other effects this has)

Notes:

• There was quite a lot of discussion previously about the impact/effect of this parameter and whether it mattered or not. Speak to Sebastian to get the information on this.
• It will be important for us to iterate the investigation of this and not get to deep. The ultimate goal is to set this to the smallest number that works.

[henrikstoklandberg]

Below it plots of the crest heigh example with the same seed, error and sem tolerance = 0.01(1%), but changing the num_simulations_per_points between. runs. There seems to be a trend that at least down to 20 simulations per point the effect of the doe seem to be more and more beneficial. However I tried to run with 10 and it took a lot of time, so I did not complete. This could however have something to do with how the current code(as of 24-06-25) is first running the sobol until convergence, then doe until convergence. So if sobol is never/taking very long to reach the given stopping criteria, this might be the reason.
20:

30:

40:

50:

100: