[BUG] Spark application stuck on installing extensions to enable rapids GPU SQL support

[Niharikadutta]

Describe the bug
I am trying to run a Spark application on a single GPU enabled VM using the spark-rapids plugin, and after submitting the application I see in the logs it gets stuck at Installing extensions to enable rapids GPU SQL support and doesn't proceed from there, for over 30 minutes. I have tried cancelling and re-submitting but always see the same behavior.

Steps/Code to reproduce bug
Here is what my spark-submit looks like:

spark-submit --master local --conf spark.plugins=com.nvidia.spark.SQLPlugin \
--conf spark.rapids.sql.incompatibleOps.enabled=true \
--conf spark.executor.cores=1 \
--conf spark.executor.resource.gpu.amount=1 --conf spark.task.resource.gpu.amount=1 \
--conf spark.executor.resource.gpu.discoveryScript=./getGpusResources.sh \
--jars /path/to/cudf.jar,\
/path/to/rapids-spark.jar \
--class ClassName /path/to/application.jar \
/path/to/application/arguments.json

Here is the stacktrace till it stops proceeding:

2021-01-20 02:40:16,750 INFO plugin.ExecutorPluginContainer: Initialized executor component for plugin com.nvidia.spark.SQLPlugin.                                          
2021-01-20 02:40:16,757 INFO util.Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 33023.                             
2021-01-20 02:40:16,757 INFO netty.NettyBlockTransferService: Server created on nidutta-ncast4v3-vm.internal.cloudapp.net:33023                                             
2021-01-20 02:40:16,759 INFO storage.BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy                                 
2021-01-20 02:40:16,766 INFO storage.BlockManagerMaster: Registering BlockManager BlockManagerId(driver, nidutta-ncast4v3-vm.internal.cloudapp.net, 33023, None)            
2021-01-20 02:40:16,768 INFO storage.BlockManagerMasterEndpoint: Registering block manager nidutta-ncast4v3-vm.internal.cloudapp.net:33023 with 2.8 GiB RAM, BlockManagerId(driver, nidutta-ncast4v3-vm.internal.cloudapp.net, 33023, None)                                                                                                             
2021-01-20 02:40:16,770 INFO storage.BlockManagerMaster: Registered BlockManager BlockManagerId(driver, nidutta-ncast4v3-vm.internal.cloudapp.net, 33023, None)             
2021-01-20 02:40:16,771 INFO storage.BlockManager: Initialized BlockManager: BlockManagerId(driver, nidutta-ncast4v3-vm.internal.cloudapp.net, 33023, None)                 
2021-01-20 02:40:16,782 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@24fba488{/metrics/json,null,AVAILABLE,@Spark}                                    
2021-01-20 02:40:16,815 WARN rapids.SQLExecPlugin: Installing extensions to enable rapids GPU SQL support. To disable GPU support set `spark.rapids.sql.enabled` to false

Expected behavior
The Spark application to run successfully, or at least proceed beyond the installing extensions step.

Environment details

• Environment location: Standalone Spark on a single GPU enabled VM
• OS: Ubuntu 18.04
• Cuda version: 10.2
• Nvidia driver version: 460.32.03
• Spark version: 3.0.1
• Spark configuration settings related to the issue (shown in spark-submit command above)

Additional context
If I remove all the spark rapids related configuration and jars, the application runs fine.

[tgravescs]

spark.executor.resource.gpu.amount=1 --conf spark.task.resource.gpu.amount=1

GPU scheduling is not supported in Spark in local mode. Please remove those configs and it should continue.

For example command see: https://nvidia.github.io/spark-rapids/docs/get-started/getting-started-on-prem.html#local-mode

[Niharikadutta]

Thanks @tgravescs for the suggestion, it worked! I just had one naïve question. How can one know for sure the GPU acceleration is actually being enabled to process the Spark application? Is there something I can lookout for in the physical plan through explain()?

[tgravescs]

yes, the best thing to do is look at the explain output or the sql UI page (the sql nodes and plan are there as well).

you can find some more information here: https://nvidia.github.io/spark-rapids/docs/get-started/getting-started-on-prem.html#monitoring

you can also look at the Debugging section right below that for it to print what is on GPU and what is not and why.

There is also a simple example join here:
https://nvidia.github.io/spark-rapids/docs/get-started/getting-started-on-prem.html#example-join-operation

Let me know if you have other questions.

[Niharikadutta]

Great thank you @tgravescs ! I am going to close this issue now.