[BUG] java.lang.UnsupportedOperationException: Type NullType not supported

[LIN-Yu-Ting]

Describe the bug
We are executing a simple Spark SQL, such as SELECT * FROM VIEW LIMIT 1000 and we are able to obtain the results.

However, when we encapsulate this SELECT * FROM VIEW command as

SELECT ARRAY( ) as PointMut,* FROM
  (
  SELECT * FROM VIEW
  )

Then exception comes. Please check the detail exception.

06:46:52.872 ERROR SparkExecuteStatementOperation - Error executing query with b400b11a-0d4f-4d54-bd39-9ccc500047c1, currentState RUNNING, 
java.lang.UnsupportedOperationException: Type NullType not supported
        at com.nvidia.spark.rapids.GpuRowToColumnConverter$.getConverterForType(GpuRowToColumnarExec.scala:147) ~[rapids-4-spark_2.12-cuda12.jar:?]
        at com.nvidia.spark.rapids.GpuRowToColumnConverter$.getConverterForType(GpuRowToColumnarExec.scala:126) ~[rapids-4-spark_2.12-cuda12.jar:?]
        at com.nvidia.spark.rapids.GpuRowToColumnConverter.$anonfun$converters$1(GpuRowToColumnarExec.scala:39) ~[rapids-4-spark_2.12-cuda12.jar:?]
        at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:286) ~[scala-library-2.12.15.jar:?]
        at scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36) ~[scala-library-2.12.15.jar:?]
        at scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:33) ~[scala-library-2.12.15.jar:?]
        at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:198) ~[scala-library-2.12.15.jar:?]
        at scala.collection.TraversableLike.map(TraversableLike.scala:286) ~[scala-library-2.12.15.jar:?]
        at scala.collection.TraversableLike.map$(TraversableLike.scala:279) ~[scala-library-2.12.15.jar:?]
        at scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:198) ~[scala-library-2.12.15.jar:?]
        at com.nvidia.spark.rapids.GpuRowToColumnConverter.<init>(GpuRowToColumnarExec.scala:38) ~[rapids-4-spark_2.12-cuda12.jar:?]
        at com.nvidia.spark.rapids.GpuRowToColumnarExec.internalDoExecuteColumnar(GpuRowToColumnarExec.scala:913) ~[rapids-4-spark_2.12-cuda12.jar:?]
        at com.nvidia.spark.rapids.GpuExec.doExecuteColumnar(GpuExec.scala:428) ~[rapids-4-spark_2.12-cuda12.jar:?]
        at com.nvidia.spark.rapids.GpuExec.doExecuteColumnar$(GpuExec.scala:426) ~[rapids-4-spark_2.12-cuda12.jar:?]
        at com.nvidia.spark.rapids.GpuRowToColumnarExec.doExecuteColumnar(GpuRowToColumnarExec.scala:861) ~[rapids-4-spark_2.12-cuda12.jar:?]
        at org.apache.spark.sql.execution.SparkPlan.$anonfun$executeColumnar$1(SparkPlan.scala:221) ~[spark-sql_2.12-3.3.0.jar:3.3.0]
        at org.apache.spark.sql.execution.SparkPlan.$anonfun$executeQuery$1(SparkPlan.scala:232) ~[spark-sql_2.12-3.3.0.jar:3.3.0]
        at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151) ~[spark-core_2.12-3.3.0.jar:3.3.0]
        at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:229) ~[spark-sql_2.12-3.3.0.jar:3.3.0]
        at org.apache.spark.sql.execution.SparkPlan.executeColumnar(SparkPlan.scala:217) ~[spark-sql_2.12-3.3.0.jar:3.3.0]
        at com.nvidia.spark.rapids.GpuBaseLimitExec.internalDoExecuteColumnar(limit.scala:151) ~[rapids-4-spark_2.12-cuda12.jar:?]
        at com.nvidia.spark.rapids.GpuBaseLimitExec.internalDoExecuteColumnar$(limit.scala:150) ~[rapids-4-spark_2.12-cuda12.jar:?]
        at com.nvidia.spark.rapids.GpuLocalLimitExec.internalDoExecuteColumnar(limit.scala:168) ~[rapids-4-spark_2.12-cuda12.jar:?]
        at com.nvidia.spark.rapids.GpuExec.doExecuteColumnar(GpuExec.scala:428) ~[rapids-4-spark_2.12-cuda12.jar:?]
        at com.nvidia.spark.rapids.GpuExec.doExecuteColumnar$(GpuExec.scala:426) ~[rapids-4-spark_2.12-cuda12.jar:?]
        at com.nvidia.spark.rapids.GpuLocalLimitExec.doExecuteColumnar(limit.scala:168) ~[rapids-4-spark_2.12-cuda12.jar:?]
        at org.apache.spark.sql.execution.SparkPlan.$anonfun$executeColumnar$1(SparkPlan.scala:221) ~[spark-sql_2.12-3.3.0.jar:3.3.0]
        at org.apache.spark.sql.execution.SparkPlan.$anonfun$executeQuery$1(SparkPlan.scala:232) ~[spark-sql_2.12-3.3.0.jar:3.3.0]
        at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151) ~[spark-core_2.12-3.3.0.jar:3.3.0]
        at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:229) ~[spark-sql_2.12-3.3.0.jar:3.3.0]
        at org.apache.spark.sql.execution.SparkPlan.executeColumnar(SparkPlan.scala:217) ~[spark-sql_2.12-3.3.0.jar:3.3.0]
        at com.nvidia.spark.rapids.GpuUnionExec.$anonfun$internalDoExecuteColumnar$17(basicPhysicalOperators.scala:1256) ~[rapids-4-spark_2.12-cuda12.jar:?]
        at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:286) ~[scala-library-2.12.15.jar:?]
        at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62) ~[scala-library-2.12.15.jar:?]
        at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55) ~[scala-library-2.12.15.jar:?]
        at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49) ~[scala-library-2.12.15.jar:?]
        at scala.collection.TraversableLike.map(TraversableLike.scala:286) ~[scala-library-2.12.15.jar:?]
        at scala.collection.TraversableLike.map$(TraversableLike.scala:279) ~[scala-library-2.12.15.jar:?]
        at scala.collection.AbstractTraversable.map(Traversable.scala:108) ~[scala-library-2.12.15.jar:?]
        at com.nvidia.spark.rapids.GpuUnionExec.internalDoExecuteColumnar(basicPhysicalOperators.scala:1256) ~[rapids-4-spark_2.12-cuda12.jar:?]
        at com.nvidia.spark.rapids.GpuExec.doExecuteColumnar(GpuExec.scala:428) ~[rapids-4-spark_2.12-cuda12.jar:?]
        at com.nvidia.spark.rapids.GpuExec.doExecuteColumnar$(GpuExec.scala:426) ~[rapids-4-spark_2.12-cuda12.jar:?]
        at com.nvidia.spark.rapids.GpuUnionExec.doExecuteColumnar(basicPhysicalOperators.scala:1228) ~[rapids-4-spark_2.12-cuda12.jar:?]
        at org.apache.spark.sql.execution.SparkPlan.$anonfun$executeColumnar$1(SparkPlan.scala:221) ~[spark-sql_2.12-3.3.0.jar:3.3.0]
        at org.apache.spark.sql.execution.SparkPlan.$anonfun$executeQuery$1(SparkPlan.scala:232) ~[spark-sql_2.12-3.3.0.jar:3.3.0]
        at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151) ~[spark-core_2.12-3.3.0.jar:3.3.0]
        at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:229) ~[spark-sql_2.12-3.3.0.jar:3.3.0]
        at org.apache.spark.sql.execution.SparkPlan.executeColumnar(SparkPlan.scala:217) ~[spark-sql_2.12-3.3.0.jar:3.3.0]
        at com.nvidia.spark.rapids.GpuColumnarToRowExec.doExecute(GpuColumnarToRowExec.scala:365) ~[rapids-4-spark_2.12-cuda12.jar:?]
        at org.apache.spark.sql.execution.SparkPlan.$anonfun$execute$1(SparkPlan.scala:194) ~[spark-sql_2.12-3.3.0.jar:3.3.0]
        at org.apache.spark.sql.execution.SparkPlan.$anonfun$executeQuery$1(SparkPlan.scala:232) ~[spark-sql_2.12-3.3.0.jar:3.3.0]
        at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151) ~[spark-core_2.12-3.3.0.jar:3.3.0]
        at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:229) ~[spark-sql_2.12-3.3.0.jar:3.3.0]
        at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:190) ~[spark-sql_2.12-3.3.0.jar:3.3.0]
        at org.apache.spark.sql.execution.SparkPlan.getByteArrayRdd(SparkPlan.scala:340) ~[spark-sql_2.12-3.3.0.jar:3.3.0]
        at org.apache.spark.sql.execution.SparkPlan.executeTake(SparkPlan.scala:473) ~[spark-sql_2.12-3.3.0.jar:3.3.0]
        at org.apache.spark.sql.execution.SparkPlan.executeTake(SparkPlan.scala:459) ~[spark-sql_2.12-3.3.0.jar:3.3.0]
        at org.apache.spark.sql.execution.CollectLimitExec.executeCollect(limit.scala:48) ~[spark-sql_2.12-3.3.0.jar:3.3.0]
        at org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanExec.$anonfun$executeCollect$1(AdaptiveSparkPlanExec.scala:340) ~[spark-sql_2.12-3.3.0.jar:3.3.0]
        at org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanExec.withFinalPlanUpdate(AdaptiveSparkPlanExec.scala:368) ~[spark-sql_2.12-3.3.0.jar:3.3.0]
        at org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanExec.executeCollect(AdaptiveSparkPlanExec.scala:340) ~[spark-sql_2.12-3.3.0.jar:3.3.0]
        at org.apache.spark.sql.Dataset.collectFromPlan(Dataset.scala:3868) ~[spark-sql_2.12-3.3.0.jar:3.3.0]
        at org.apache.spark.sql.Dataset.$anonfun$collect$1(Dataset.scala:3120) ~[spark-sql_2.12-3.3.0.jar:3.3.0]
        at org.apache.spark.sql.Dataset.$anonfun$withAction$2(Dataset.scala:3858) ~[spark-sql_2.12-3.3.0.jar:3.3.0]
        at org.apache.spark.sql.execution.QueryExecution$.withInternalError(QueryExecution.scala:510) ~[spark-sql_2.12-3.3.0.jar:3.3.0]
        at org.apache.spark.sql.Dataset.$anonfun$withAction$1(Dataset.scala:3856) ~[spark-sql_2.12-3.3.0.jar:3.3.0]
        at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$6(SQLExecution.scala:109) ~[spark-sql_2.12-3.3.0.jar:3.3.0]
        at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:169) ~[spark-sql_2.12-3.3.0.jar:3.3.0]
        at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:95) ~[spark-sql_2.12-3.3.0.jar:3.3.0]
        at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:779) ~[spark-sql_2.12-3.3.0.jar:3.3.0]
        at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:64) ~[spark-sql_2.12-3.3.0.jar:3.3.0]
        at org.apache.spark.sql.Dataset.withAction(Dataset.scala:3856) ~[spark-sql_2.12-3.3.0.jar:3.3.0]
        at org.apache.spark.sql.Dataset.collect(Dataset.scala:3120) ~[spark-sql_2.12-3.3.0.jar:3.3.0]
        at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation.org$apache$spark$sql$hive$thriftserver$SparkExecuteStatementOperation$$execute(SparkExecuteStatementOperation.scala:300) ~[spark-hive-thriftserver_2.12-3.3.0.jar:3.3.0]
        at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation$$anon$2$$anon$3.$anonfun$run$2(SparkExecuteStatementOperation.scala:230) ~[spark-hive-thriftserver_2.12-3.3.0.jar:3.3.0]
        at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) ~[scala-library-2.12.15.jar:?]
        at org.apache.spark.sql.hive.thriftserver.SparkOperation.withLocalProperties(SparkOperation.scala:79) ~[spark-hive-thriftserver_2.12-3.3.0.jar:3.3.0]
        at org.apache.spark.sql.hive.thriftserver.SparkOperation.withLocalProperties$(SparkOperation.scala:63) ~[spark-hive-thriftserver_2.12-3.3.0.jar:3.3.0]
        at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation.withLocalProperties(SparkExecuteStatementOperation.scala:43) ~[spark-hive-thriftserver_2.12-3.3.0.jar:3.3.0]
        at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation$$anon$2$$anon$3.run(SparkExecuteStatementOperation.scala:230) ~[spark-hive-thriftserver_2.12-3.3.0.jar:3.3.0]
        at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation$$anon$2$$anon$3.run(SparkExecuteStatementOperation.scala:225) ~[spark-hive-thriftserver_2.12-3.3.0.jar:3.3.0]
        at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_442]
        at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_442]
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1878) ~[hadoop-client-api-3.3.2.jar:?]
        at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation$$anon$2.run(SparkExecuteStatementOperation.scala:239) ~[spark-hive-thriftserver_2.12-3.3.0.jar:3.3.0]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_442]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_442]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_442]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_442]
        at java.lang.Thread.run(Thread.java:750) ~[?:1.8.0_442]

Steps/Code to reproduce bug
Please provide a list of steps or a code sample to reproduce the issue.
Avoid posting private or sensitive data.

Expected behavior
A clear and concise description of what you expected to happen.

Environment details (please complete the following information)

• Environment location: [Standalone, Kubernetes, Cloud(Azure)]
• Spark 3.3.0
• Spark configuration settings related to the issue

Additional context
We are using newest Spark Rapids version v25.02.1.

[gerashegalov]

Hi @LIN-Yu-Ting , we need more details to reproduce the issue. Plugin version, Spark version, schema, data format, etc.

I did a smoke test without HiveThriftServer, Spark 3.5.5, 25.02.1 which works for me

from pyspark.sql import SparkSession
from pprint import pprint

spark = SparkSession.builder.getOrCreate()
spark.range(100).write.mode("overwrite").saveAsTable("RANGE100_TABLE")
spark.sql("CREATE OR REPLACE VIEW RANGE50_VIEW AS SELECT * FROM RANGE100_TABLE WHERE id < 50")
result = spark.sql("SELECT ARRAY(), * FROM RANGE50_VIEW").collect()
pprint(result)

Then running

TZ=UTC ~/dist/spark-3.5.5-bin-hadoop3-scala2.13/bin/spark-submit --packages com.nvidia:rapids-4-spark_2.13:25.02.1 --conf spark.plugins=com.nvidia.spark.SQLPlugin --conf spark.rapids.sql.explain=ALL repro.py 

25/04/10 15:00:09 WARN GpuOverrides: 
*Exec <ProjectExec> will run on GPU
  *Expression <Alias> [] AS array()#6 will run on GPU
  *Exec <FilterExec> will run on GPU
    *Expression <And> (isnotnull(id#3L) AND (id#3L < 50)) will run on GPU
      *Expression <IsNotNull> isnotnull(id#3L) will run on GPU
      *Expression <LessThan> (id#3L < 50) will run on GPU
    *Exec <FileSourceScanExec> will run on GPU
[Row(array()=[], id=43),
 Row(array()=[], id=44),
 Row(array()=[], id=45),
...

[LIN-Yu-Ting]

@gerashegalov Thanks for your response.

We only found this exception in case of using HiveThriftServer. If we execute the same command in PySpark then no more exception. Furthermore, in order to narrow down the possible root cause, I have simplified our command as following.

However, it seems that the exception is related to our HiveUDF function as If I removed it then no more exception appears.
Our HiveUDF is written in Java and do not support GPU yet. Normally, it should roll back calculation to CPU. Is there any possibility that null value is causing some error while converting values between CPU and GPU.

This is the schema of variants table:

root
 |-- annoGeneArr: array (nullable = true)
 |    |-- element: string (containsNull = true)
 |-- Consequence: string (nullable = true)
 |-- Functional Effect: string (nullable = true)
 |-- annoConseqArr: array (nullable = true)
 |    |-- element: string (containsNull = true)

[revans2]

In this case I think the fix is simpler than the repro case. NullType is not common, but it is not hard to make something a NullType by just adding a null column. spark.range(100).selectExpr("null as n") n is now null type, but spark also aggressively optimizes away the null when you try to do anything with it. If you have a repro case, then we can just update https://github.com/NVIDIA/spark-rapids/blob/branch-25.06/sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuRowToColumnarExec.scala#L97-L149 to include a NullType match, where the converter just inserts in nulls for every row it sees into an INT8 column. The underlying data type we use for NullType is INT8.

[LIN-Yu-Ting]

@revans2 I can provide data to reproduce this issue. However, as it seems that this exception is related to our UDF function. Do we need to provide this UDF function or not ?

[revans2]

@LIN-Yu-Ting

It might be. From the stack trace all I can see is

GpuRowToColumnConverter
GpuLocalLimitExec
GpuUnionExec
GpuColumnarToRowExec
CollectLimitExec

But this is what came after the point where the exception happened. If the plan had a UDF just before the row to columnar conversion, then it might be triggering it. Our code has different optimizations for different data types when it comes to column to row and row to column conversions. What I really would like to see is the plan that Spark is trying to run and the schema of the plan at the point that conversion happens. If you don't fee comfortable posting it we can work with you to send it other ways.

[igorpeshansky]

This should be fixed by #14669, which was released as part of 26.06+. @LIN-Yu-Ting, are you able to re-run your query with a newer cuDF plugin for Spark release? Please reopen if the crash is still occurring with 26.06.1.