GPU slurm settings

[cbegeman]

Let's discuss the ideal number of nodes/threads for GPU tests. @mwarusz mentioned in particular that the number of CPU processors per cell count is non-optimal for GPU tests.

[cbegeman]

Also, @mwarusz mentioned that this change is needed (correct me if I misunderstood):

Add SBATCH --gpus-per-task=1 here

polaris/polaris/job/job_script.slurm.template

Line 20 in ed1e0d5

#SBATCH --gpus-per-node={{ gpus_per_node }}

[mwarusz]

You also want to limit the total number of MPI processes per node to not exceed gpus_per_node.

[mwarusz]

For example, on pm-gpu I think we want to follow this example settings from NERSC: https://docs.nersc.gov/systems/perlmutter/running-jobs/#4-nodes-16-tasks-16-gpus-1-gpu-visible-to-each-task

[mwarusz]

The number of cells per gpu should ideally be above 8000.

[xylar]

Thanks for starting this discussion. I have already made some progress on this as part of adding aurora GPU support but had only realized the problem with cells per gpu, not thought it through. This will take some somewhat significant work in both mache and Polaris to implement but it's a high priority for me when I get back next week.

[xylar]

@cbegeman and @mwarusz, can you verify if this is addressed for you in Polaris, following #474?