[AutoSparkUT] [SPARK-31284: compatibility with Spark 2.4 in reading timestamps] - GPU Execution Issue

[wjxiz1992]

Describe the bug

The GPU ORC reader returns the wrong pre-Gregorian timestamp when reading the
Spark 2.4 compatibility resource. Spark CPU returns
1001-01-01 01:02:03.123456; GPU returns
1001-01-07 01:09:05.123456. The mismatch reproduces in V1 and V2 with both
row and vectorized ORC reader settings.

Steps/code to reproduce

Build the exact PR head, then use Spark 3.3.0 and the complete RAPIDS UT
configuration:

mvn package -DskipTests -pl dist -am -Dbuildver=330 \
  -Dmaven.repo.local=./.mvn-repo

$SPARK_HOME/bin/spark-shell \
  --master local[2] \
  --conf spark.sql.optimizer.excludedRules=org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation,org.apache.spark.sql.catalyst.optimizer.ConstantFolding \
  --conf spark.rapids.sql.enabled=true \
  --conf spark.plugins=com.nvidia.spark.SQLPlugin \
  --conf spark.sql.queryExecutionListeners=org.apache.spark.sql.rapids.ExecutionPlanCaptureCallback \
  --conf spark.rapids.sql.explain=ALL \
  --conf spark.rapids.sql.test.isFoldableNonLitAllowed=true \
  --conf spark.rapids.sql.csv.read.decimal.enabled=true \
  --conf spark.rapids.sql.format.avro.enabled=true \
  --conf spark.rapids.sql.format.avro.read.enabled=true \
  --conf spark.rapids.sql.format.hive.text.write.enabled=true \
  --conf spark.rapids.sql.format.json.enabled=true \
  --conf spark.rapids.sql.format.json.read.enabled=true \
  --conf spark.rapids.sql.incompatibleDateFormats.enabled=true \
  --conf spark.rapids.sql.python.gpu.enabled=true \
  --conf spark.rapids.sql.rowBasedUDF.enabled=true \
  --conf spark.rapids.sql.window.collectList.enabled=true \
  --conf spark.rapids.sql.window.collectSet.enabled=true \
  --conf spark.rapids.sql.window.range.byte.enabled=true \
  --conf spark.rapids.sql.window.range.short.enabled=true \
  --conf spark.rapids.sql.expression.Ascii=true \
  --conf spark.rapids.sql.expression.Conv=true \
  --conf spark.rapids.sql.expression.GetJsonObject=true \
  --conf spark.rapids.sql.expression.JsonToStructs=true \
  --conf spark.rapids.sql.expression.StructsToJson=true \
  --conf spark.rapids.sql.exec.CollectLimitExec=true \
  --conf spark.rapids.sql.exec.FlatMapCoGroupsInPandasExec=true \
  --conf spark.rapids.sql.exec.WindowInPandasExec=true \
  --conf spark.rapids.sql.hasExtendedYearValues=false \
  --conf spark.unsafe.exceptionOnMemoryLeak=true \
  --conf spark.sql.session.timeZone=America/Los_Angeles \
  --jars $RAPIDS_JAR

Run extract-test-configs.scala first. In the same session, run GPU first and
CPU second for V1/V2 and for
spark.sql.orc.enableVectorizedReader=false,true:

import java.util.TimeZone
TimeZone.setDefault(TimeZone.getTimeZone("America/Los_Angeles"))
spark.conf.set("spark.sql.orc.impl", "native")

val path = "/path/to/before_1582_ts_v2_4.snappy.orc"
val df = spark.read.orc(path)
println(df.collect().toSeq)
println(df.queryExecution.executedPlan)

Resource SHA-256:
8c070cb17cab05ac726191881a20ced5e0318dbfbf80606ce7e8517bd6edf2b2.

Actual output

RAPIDS Enabled = YES
Plugins Loaded = YES
SPARK_VERSION = 3.3.0

GPU-V1, vectorized=false: rows=[[1001-01-07 01:09:05.123456]]; plan=GpuFileGpuScan orc
GPU-V1, vectorized=true:  rows=[[1001-01-07 01:09:05.123456]]; plan=GpuFileGpuScan orc
GPU-V2, vectorized=false: rows=[[1001-01-07 01:09:05.123456]]; plan=GpuBatchScan ... GpuOrcScan
GPU-V2, vectorized=true:  rows=[[1001-01-07 01:09:05.123456]]; plan=GpuBatchScan ... GpuOrcScan

CPU-V1/V2, vectorized=false/true: rows=[[1001-01-01 01:02:03.123456]]

Expected behavior

GPU should return 1001-01-01 01:02:03.123456, matching Spark CPU's Spark 2.4
ORC calendar and timezone compatibility behavior.

Environment

• Local Spark 3.3.0, revision f74867bddfbcdd4d08076db36851e88b15e66556
• cudf-spark f7457eeee116ca033ffa84eadbe84226e01b2ff7, version
  26.10.0-SNAPSHOT, CUDA 12 JAR
• NVIDIA RTX 5880 Ada Generation, driver 580.95.05
• Java 17

Additional context

• Original Spark test: OrcSourceSuite.scala lines 512-518
• Affected migrated suites: RapidsOrcSourceV1Suite, RapidsOrcSourceV2Suite
• Migration tracker: [AutoSparkUT] Migrate ten additional Spark SQL suites #15465
• Migration PR: [AutoSparkUT] Migrate ten Spark SQL suites [fast-ut] [reduced-it] #15466
• Classification: GPU/CPU data-correctness bug; recovery priority P0
• Reproduction rate: 100% in all four V1/V2 and reader-mode combinations