[QST] Setting Exclusive_Mode on GPUs making it unavailable for ML workloads

[Niharikadutta]

What is your question?
I am running Spark on YARN without isolation and hence following the instructions in this doc by setting each GPU to EXCLUSIVE_MODE.
While this works for ETL workloads, this setting makes other ML (tensorflow, pytorch) workloads fail by giving the following error:

all CUDA-capable devices are busy or unavailable

Interestingly, this happens even if there is no active Spark job running on the cluster, which should ideally mean the GPUs are available to be used by other processes?
What is the recommended way to run ETL + ML workloads on the same YARN cluster such that GPUs can be shared?

[tgravescs]

are you using GPU scheduling on the YARN setup?
How many ML jobs are you running on a box and are they trying to share GPU's? The easiest way to check this is to run the job, go to the host where it got the error and use nvidia-smi to see if the GPUs are already allocated to other processes.
For instance if you have 4 GPUs on a box and try to have 5 processes on that box use the GPUs, then one of them will fail.

[Niharikadutta]

@tgravescs yes I'm using GPU scheduling on the YARN setup by setting following yarn configs:

<property>
      <name>yarn.resource-types</name>
      <value>yarn.io/gpu</value>
    </property>
    <property>
      <name>yarn.scheduler.capacity.resource-calculator</name>
      <value>org.apache.hadoop.yarn.util.resource.DominantResourceCalculator</value>
    </property>
    <property>
      <name>yarn.nodemanager.resource-plugins</name>
      <value>yarn.io/gpu</value>
    </property>

And setting all GPUs with exclusive_mode property.

Here is the output of nvidia-smi after spinning up a cluster but before running any job on it:

It looks like without running any active job, all GPUs are being used by java processes. Is this expected?

[tgravescs]

No, something else is using your GPUs, if its not scheduling them through yarn then they will obviously conflict and hence why you get an error about the cuda devices not available. Suggest you look to see what those processes are and if they are using GPU scheduling in yarn as well.

[viadea]

@tgravescs I checked with @Niharikadutta today and we suspect those 4 JAVA processes are Spark Executors.
@Niharikadutta will double check on this assumption.

Since currently they are using YARN without Isolation mode, so all 4 GPUs are in exclusive mode. If all Spark executors will keep running even after Spark job finishes, is there any way to run other ML jobs on GPU?
My understanding is that other ML jobs can not since all GPUs are occupied by Spark executors.

Now if @Niharikadutta switched from "YARN without Isolation" to "YARN 3.1.3 with Isolation and GPU Scheduling Enabled " mode, of course, there is no requirement to enable GPU in exclusive mode.
But do you think if other MLs jobs can share the same GPUs with long-running Spark executors?

[tgravescs]

I don't understand the situation:

Spark job finishes

What exactly do you mean by this? Is the spark application finishing or is this trying to run ML jobs within the same application? ie one application uses the sql plugin to run etc jobs and then afterwards the same applications runs ML job. Are these ML jobs python?

[tgravescs]

If the answer to the above is yes, its all in the same application and you have separate java and python process, then you couldn't do it in process exclusive mode, but if you are using yarn with isolation and gpu scheduling it would allow both to use the GPU. You would need to split the memory though across them, otherwise the rapids plugin will use all the GPU memory

[viadea]

@Niharikadutta @tgravescs As discussed today, if the ML job(Horovod on Spark) needs to run after the ETL portion has finished, we may need to split the GPU memory to 2 parts -- one for ETL job(Spark executor), one for Horovod job(which is a separate python process using GPU).

Say for a T4 with 16G GPU memory, we can allocate 6GB to Spark executor, and 10GB to other stuff(including ML: Horovod job in this case).
The 10GB can be reserved by using parameter spark.rapids.memory.gpu.reserve.

To troubleshoot if the Horovod issue is caused by inefficient GPU memory, we can try to do one test to troubleshoot:

1. Reserve most of the GPU memory for Horovod. (Say spark.rapids.memory.gpu.reserve set to 16G or 15G)
2. Disable rapids pluginn for ETL portion to let it run on CPU without using any GPU memory.
3. Run the Horovod job after ETL is done to see if this issue still exists.

If the issue still exists, then we can know this issue is not related to GPU memory allocation.

[viadea]

Another option I can think of is to:

• Either use different pools for ETL or ML(Horovod);
• Or restart the Spark executors(with different GPU memory settings spark.rapids.memory.gpu.reserve) between ETL and ML in the same pool.

[tgravescs]

you can't really restart the executors in Spark. You could potentially use stage level scheduling feature (http://spark.apache.org/docs/latest/configuration.html#stage-level-scheduling-overview) but it requires dynamic allocation. Essentially with that, you would get one set of executors to do the ETL with spark rapids plugin and then you would specify a different profile for the ML side which would tell spark to get different executors to run ML on. Note that with dynamic allocation, the ETL executors would just idle timeout when not in use.

[viadea]

Yes @tgravescs . Here "restart the Spark executors" actually means restarting the spark jobs in the same pool.

[Niharikadutta]

Closing this issue since this is by design,