Describe the bug
The GPU ORC reader cannot read valid ORC schemas whose field names consist only
of numbers. The first top-level case from Spark's SPARK-36663 test fails because
the GPU path passes struct<1:string,2:string,3:string> to the Catalyst SQL data
type parser. Spark CPU reads the row and schema successfully. V1 and V2 both
fail through the GPU ORC reader.
Steps/code to reproduce
Build the exact PR head, then use Spark 3.3.0 and the complete RAPIDS UT
configuration:
mvn package -DskipTests -pl dist -am -Dbuildver=330 \
-Dmaven.repo.local=./.mvn-repo
$SPARK_HOME/bin/spark-shell \
--master local[2] \
--conf spark.sql.optimizer.excludedRules=org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation,org.apache.spark.sql.catalyst.optimizer.ConstantFolding \
--conf spark.rapids.sql.enabled=true \
--conf spark.plugins=com.nvidia.spark.SQLPlugin \
--conf spark.sql.queryExecutionListeners=org.apache.spark.sql.rapids.ExecutionPlanCaptureCallback \
--conf spark.rapids.sql.explain=ALL \
--conf spark.rapids.sql.test.isFoldableNonLitAllowed=true \
--conf spark.rapids.sql.csv.read.decimal.enabled=true \
--conf spark.rapids.sql.format.avro.enabled=true \
--conf spark.rapids.sql.format.avro.read.enabled=true \
--conf spark.rapids.sql.format.hive.text.write.enabled=true \
--conf spark.rapids.sql.format.json.enabled=true \
--conf spark.rapids.sql.format.json.read.enabled=true \
--conf spark.rapids.sql.incompatibleDateFormats.enabled=true \
--conf spark.rapids.sql.python.gpu.enabled=true \
--conf spark.rapids.sql.rowBasedUDF.enabled=true \
--conf spark.rapids.sql.window.collectList.enabled=true \
--conf spark.rapids.sql.window.collectSet.enabled=true \
--conf spark.rapids.sql.window.range.byte.enabled=true \
--conf spark.rapids.sql.window.range.short.enabled=true \
--conf spark.rapids.sql.expression.Ascii=true \
--conf spark.rapids.sql.expression.Conv=true \
--conf spark.rapids.sql.expression.GetJsonObject=true \
--conf spark.rapids.sql.expression.JsonToStructs=true \
--conf spark.rapids.sql.expression.StructsToJson=true \
--conf spark.rapids.sql.exec.CollectLimitExec=true \
--conf spark.rapids.sql.exec.FlatMapCoGroupsInPandasExec=true \
--conf spark.rapids.sql.exec.WindowInPandasExec=true \
--conf spark.rapids.sql.hasExtendedYearValues=false \
--conf spark.unsafe.exceptionOnMemoryLeak=true \
--conf spark.sql.session.timeZone=America/Los_Angeles \
--jars $RAPIDS_JAR
Run extract-test-configs.scala first. In the same session, run GPU first and
CPU second for spark.sql.sources.useV1SourceList=orc and "", with a fresh
non-existing output path each time:
spark.sql("SELECT 'a' as `1`, 'b' as `2`, 'c' as `3`")
.write.orc(outputPath)
val df = spark.read.orc(outputPath)
println(df.collect().toSeq)
println(df.schema)
println(df.queryExecution.executedPlan)
Actual output
RAPIDS Enabled = YES
Plugins Loaded = YES
SPARK_VERSION = 3.3.0
GPU-V1/GPU-V2:
org.apache.spark.SparkException: Job aborted due to stage failure
Caused by: org.apache.spark.sql.catalyst.parser.ParseException:
Syntax error at or near '1'
== SQL ==
struct<1:string,2:string,3:string>
-------^^^
at com.nvidia.spark.rapids.SchemaUtils$.toCatalystSchema(SchemaUtils.scala:62)
at com.nvidia.spark.rapids.OrcTableReader.next(GpuOrcScan.scala:3081)
CPU-V1/CPU-V2:
rows=[[a,b,c]]
schema=StructType(
StructField(1,StringType,true),
StructField(2,StringType,true),
StructField(3,StringType,true))
Expected behavior
GPU should return the same row and schema as CPU for numeric-only top-level and
nested ORC field names.
Environment
- Local Spark 3.3.0, revision
f74867bddfbcdd4d08076db36851e88b15e66556
- cudf-spark
f7457eeee116ca033ffa84eadbe84226e01b2ff7, version
26.10.0-SNAPSHOT, CUDA 12 JAR
- NVIDIA RTX 5880 Ada Generation, driver
580.95.05
- Java 17
Additional context
Describe the bug
The GPU ORC reader cannot read valid ORC schemas whose field names consist only
of numbers. The first top-level case from Spark's SPARK-36663 test fails because
the GPU path passes
struct<1:string,2:string,3:string>to the Catalyst SQL datatype parser. Spark CPU reads the row and schema successfully. V1 and V2 both
fail through the GPU ORC reader.
Steps/code to reproduce
Build the exact PR head, then use Spark 3.3.0 and the complete RAPIDS UT
configuration:
Run
extract-test-configs.scalafirst. In the same session, run GPU first andCPU second for
spark.sql.sources.useV1SourceList=orcand"", with a freshnon-existing output path each time:
Actual output
Expected behavior
GPU should return the same row and schema as CPU for numeric-only top-level and
nested ORC field names.
Environment
f74867bddfbcdd4d08076db36851e88b15e66556f7457eeee116ca033ffa84eadbe84226e01b2ff7, version26.10.0-SNAPSHOT, CUDA 12 JAR580.95.05Additional context
OrcSourceSuite.scalalines 655-829RapidsOrcSourceV1Suite,RapidsOrcSourceV2Suitecd1b7e1777f; the GPUSchemaUtilspath still parses the unquoted ORC schema string.