More Compute Nodes are not allocating for an MPI job

[gpkmohan]

Hi List

I am trying to do a n MPI calculation and It was running ok with 32 CPUs as shown in the Slurm script below

#!/bin/bash

#SBATCH --time=20:05:00
#SBATCH --partition=compute # important
#SBATCH --nodes=4 # NUMBER OF NODES
#SBATCH --ntasks-per-node=8 # n cores / node

mpiexec -n 32 /home/dr_preethibaselios_gmail_com/qestatic/bin/pw.x < scftest.pw > log 2>&1

I use 8-core compute nodes (C2 type with 8 cores) and I have sufficient quotas in .

But when I use,
nodes=8, ntasks-per-node=8 , and mpiexec -n 64 ...
It was NOT allocating 8 nodes but 7 nodes only!

And it can seen from my Dashboard () and
[dr_preethibaselios_gmail_com@hpcslurm-slurm-login-001 testqe]$ sinfo

PARTITION AVAIL  TIMELIMIT  NODES  STATE NODELIST
compute      up   infinite      1 alloc# hpcslurm-computenodeset-0
compute      up   infinite      7 alloc! hpcslurm-computenodeset-[1-7]
compute      up   infinite     12  idle~ hpcslurm-computenodeset-[8-19]
debug*       up   infinite      4  idle~ hpcslurm-debugnodeset-[0-3]
h3           up   infinite     20  idle~ hpcslurm-h3nodeset-[0-19]

[dr_preethibaselios_gmail_com@hpcslurm-slurm-login-001 testqe]$ squeue 
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
                67   compute sub_n-8_ dr_preet CF       3:01      8 hpcslurm-computenodeset-[0-7]

My question is: Why is not allocating, then only it make a 8-node cluster?
How to check that I am necessary quotas other than instances to run this job?

Thanks in advance

Km

[mr0re1]

Hello @gpkmohan ,

Why is not allocating, then only it make a 8-node cluster?

squeue clearly shows that 8 nodes (hpcslurm-computenodeset-[0-7]) were "allocated" (i.e. assigned) to job #67.
Is your concern that nodes hpcslurm-computenodeset-[0-7] don't start up properly to execute the job?

Could you please take a look at /slurm/scripts/log/resume.log to check if there are any errors?

[mr0re1]

For information about required quota it's hard to say without seeing your blueprint.
If you use one our examples, please refer to https://github.com/GoogleCloudPlatform/hpc-toolkit/blob/main/examples/README.md#quota-requirements-for-hpc-slurmyaml

[gpkmohan]

Dear Ivan
Thanks for the mail and info. It was fixed by increasing quota-

by the way, I modified variables from BEFORE
./ghpc deploy hpc-slurm , and it was:
c2-standard-60 -> c2-standard-16
with primary/modules/embedded/community/modules/compute/schedmd-slurm-gcp-v6-nodeset/variable.tf , and

disk_size_gb = 50 -> disk_size_gb = 100
by modifying:

variable "disk_size_gb" {
  type        = number
  description = "Boot disk size in GB."
  default     = 100  <-- here we change, see below File
}

in [primary/modules/embedded/community/modules/scheduler/schedmd-slurm-gcp-v6-login/variables.tf ]

And it was all OK with increasing quota ( I chose this since quota increase request for c2-standard-16 machines are not yet a problem, but If I increase c2-standard-60, it may arise some issues, I guess)

cheers