Skip to content

[AutoSparkUT] [SPARK-36663: OrcUtils.toCatalystSchema should correctly handle a column name which consists of only numbers] - GPU Execution Issue #15472

Description

@wjxiz1992

Describe the bug

The GPU ORC reader cannot read valid ORC schemas whose field names consist only
of numbers. The first top-level case from Spark's SPARK-36663 test fails because
the GPU path passes struct<1:string,2:string,3:string> to the Catalyst SQL data
type parser. Spark CPU reads the row and schema successfully. V1 and V2 both
fail through the GPU ORC reader.

Steps/code to reproduce

Build the exact PR head, then use Spark 3.3.0 and the complete RAPIDS UT
configuration:

mvn package -DskipTests -pl dist -am -Dbuildver=330 \
  -Dmaven.repo.local=./.mvn-repo

$SPARK_HOME/bin/spark-shell \
  --master local[2] \
  --conf spark.sql.optimizer.excludedRules=org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation,org.apache.spark.sql.catalyst.optimizer.ConstantFolding \
  --conf spark.rapids.sql.enabled=true \
  --conf spark.plugins=com.nvidia.spark.SQLPlugin \
  --conf spark.sql.queryExecutionListeners=org.apache.spark.sql.rapids.ExecutionPlanCaptureCallback \
  --conf spark.rapids.sql.explain=ALL \
  --conf spark.rapids.sql.test.isFoldableNonLitAllowed=true \
  --conf spark.rapids.sql.csv.read.decimal.enabled=true \
  --conf spark.rapids.sql.format.avro.enabled=true \
  --conf spark.rapids.sql.format.avro.read.enabled=true \
  --conf spark.rapids.sql.format.hive.text.write.enabled=true \
  --conf spark.rapids.sql.format.json.enabled=true \
  --conf spark.rapids.sql.format.json.read.enabled=true \
  --conf spark.rapids.sql.incompatibleDateFormats.enabled=true \
  --conf spark.rapids.sql.python.gpu.enabled=true \
  --conf spark.rapids.sql.rowBasedUDF.enabled=true \
  --conf spark.rapids.sql.window.collectList.enabled=true \
  --conf spark.rapids.sql.window.collectSet.enabled=true \
  --conf spark.rapids.sql.window.range.byte.enabled=true \
  --conf spark.rapids.sql.window.range.short.enabled=true \
  --conf spark.rapids.sql.expression.Ascii=true \
  --conf spark.rapids.sql.expression.Conv=true \
  --conf spark.rapids.sql.expression.GetJsonObject=true \
  --conf spark.rapids.sql.expression.JsonToStructs=true \
  --conf spark.rapids.sql.expression.StructsToJson=true \
  --conf spark.rapids.sql.exec.CollectLimitExec=true \
  --conf spark.rapids.sql.exec.FlatMapCoGroupsInPandasExec=true \
  --conf spark.rapids.sql.exec.WindowInPandasExec=true \
  --conf spark.rapids.sql.hasExtendedYearValues=false \
  --conf spark.unsafe.exceptionOnMemoryLeak=true \
  --conf spark.sql.session.timeZone=America/Los_Angeles \
  --jars $RAPIDS_JAR

Run extract-test-configs.scala first. In the same session, run GPU first and
CPU second for spark.sql.sources.useV1SourceList=orc and "", with a fresh
non-existing output path each time:

spark.sql("SELECT 'a' as `1`, 'b' as `2`, 'c' as `3`")
  .write.orc(outputPath)

val df = spark.read.orc(outputPath)
println(df.collect().toSeq)
println(df.schema)
println(df.queryExecution.executedPlan)

Actual output

RAPIDS Enabled = YES
Plugins Loaded = YES
SPARK_VERSION = 3.3.0

GPU-V1/GPU-V2:
org.apache.spark.SparkException: Job aborted due to stage failure
Caused by: org.apache.spark.sql.catalyst.parser.ParseException:
Syntax error at or near '1'

== SQL ==
struct<1:string,2:string,3:string>
-------^^^

at com.nvidia.spark.rapids.SchemaUtils$.toCatalystSchema(SchemaUtils.scala:62)
at com.nvidia.spark.rapids.OrcTableReader.next(GpuOrcScan.scala:3081)

CPU-V1/CPU-V2:
rows=[[a,b,c]]
schema=StructType(
  StructField(1,StringType,true),
  StructField(2,StringType,true),
  StructField(3,StringType,true))

Expected behavior

GPU should return the same row and schema as CPU for numeric-only top-level and
nested ORC field names.

Environment

  • Local Spark 3.3.0, revision f74867bddfbcdd4d08076db36851e88b15e66556
  • cudf-spark f7457eeee116ca033ffa84eadbe84226e01b2ff7, version
    26.10.0-SNAPSHOT, CUDA 12 JAR
  • NVIDIA RTX 5880 Ada Generation, driver 580.95.05
  • Java 17

Additional context

Metadata

Metadata

Assignees

No one assigned

    Labels

    ? - Needs TriageNeed team to review and classifybugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions