[QST] Number of Tasks per Executor question

[eyalhir74]

I'm probably missing something but..
I have a V100 connected to 24 cpu cores (a K8 POD) and try to run the following configuration, which I thought would use 5 CPU cores per task, running four concurrent streams:

$SPARK_HOME/bin/spark-shell --master "local[*]" --driver-memory 50g --conf spark.locality.wait=0s --conf spark.sql.files.maxPartitionBytes=2048m --conf spark.hadoop.parquet.writer.max-padding=0 --conf spark.hadoop.parquet.block.size=268435456 --conf spark.hadoop.parquet.enable.statistics=false --conf spark.rapids.memory.pinnedPool.size=30G --conf spark.plugins=com.nvidia.spark.SQLPlugin --jars ${SPARK_CUDF_JAR},${SPARK_RAPIDS_PLUGIN_JAR} --conf spark.rapids.sql.concurrentGpuTasks=4 --conf spark.executor.cores=20 --conf spark.executor.resource.gpu.amount=1 --conf spark.task.resource.gpu.amount=0.25 --conf spark.task.cpus=5

However when I try to run the following command, it just hangs (without the amount/task.cpus etc, it runs just fine):
scala> val df = spark.read.parquet("hdfs://test/BigFile-1") [Stage 0:> (0 + 0) / 1]
Note the 0+0 setting.

Thanks.

[tgravescs]

you are running local mode, Spark doesn't support GPU scheduling in local mode. Remove all the "*resource.gpu*" configs. You can still use the GPU you just can't ask Spark to schedule it.

[jlowe]

The Spark scheduler is hanging because it cannot allocate the required resources for the executor. The configs are asking for each executor to have a GPU and each task to have 1/4 of a GPU, but no configs were specified on how Spark could locate any GPUs (i.e.: a GPU resource discovery script). See the Apache Spark documentation on custom resource scheduling for details.

If you only have one GPU then since you're running in local mode I would recommend removing the spark.executor.resource.gpu.amount and spark.task.resource.gpu.amount configs. If you have more than one GPU then you'll want to specify a GPU resource discovery script so the Spark scheduler can locate the requested GPU for the executor.

[eyalhir74]

Thank you!
And what about those: --conf spark.rapids.sql.concurrentGpuTasks=4 --conf spark.executor.cores=24 --conf spark.task.cpus=6

Should I only leave the concurrentGpuTasks and it would be suffice for concurrent threads? I believe I do see concurrency in the profiler that way

[jlowe]

Yes, those configs still apply in local mode and will affect how tasks are scheduled. Whether 4 is a good value for the concurrent GPU tasks setting highly depends on the query you're running and the amount of GPU memory available.

[jlowe]

@eyalhir74 do you have further questions on this topic or can this issue be closed?