potential overcommit.

[EricDeveaud]

Hello

dfast satndalone version may present a potential overcommit when used via slurm

dfc/dev/gff/gff_config.py and dfc/default_config.py uses multiprocessing.cpu_count() to ge the value of avilalble cpus.
multiprocessing.cpu_count() returns the number of available cpu, but this is not the same as the number of cpu available to the process. For example, you can run in a taskset context or a batch scheduler like slurm.

see:

$ nproc
96
$ taskset -c 1 nproc
1
$ taskset -c 1 python3 -c "import multiprocessing; print(multiprocessing.cpu_count())"
96

I would suggest to use len(os.sched_getaffinity(0)) instead of multiprocessing.cpu_count()

$ python3 -c "import os; print(len(os.sched_getaffinity(0)))"
96
$ taskset -c 1 python3 -c "import os; print(len(os.sched_getaffinity(0)))"
1

regards

Eric

[nigyta]

Thanks for the suggestion.
I'll consider using os.sched_getaffinity in the next update.

Currently, DFAST will use max(available CPU cores, 4 by default, unless specified with --cpu option.
I recommend using --cpu option when running it on a HPC cluster.

[EricDeveaud]

thanks for taking it in consideration.

having a mechanism to "coerce" tha max cpu useable, is a security for us. One may forgot the value set in config, or missues the --cpu option.

maybee if requested cpus > availables one, a warning message may help

regards

Eric