Support DST timezones conversion for ORC [databricks]

[res-life]

Fixes #13437.

Depends on:

• part 1: [Part1] [ORC-timezone]: Build ORC timezone metadata at runtime, drop pregenerated binary cudf-spark-jni#4539
• part 2: [Part 2] [ORC-timezone]: Extract recurring DST rule from JVM timezone APIs cudf-spark-jni#4635
• part 3: [Part 3] [ORC-timezone]: GPU kernel for DST-aware timezone conversion cudf-spark-jni#4733
• ORC 2015 base-offset borrow fix: Fix ORC base-offset borrow cudf-spark-jni#4809
• part 4: [Part 4] [ORC-timezone]: Wire recurring DST metadata through JNI cudf-spark-jni#4812
• part 5: [Part 5] [ORC-timezone]: Enable DST-aware ORC timezone conversion cudf-spark-jni#4813

Description

Context

ORC OSS uses java.util.TimeZone to do rebase, it does not use java.time.ZoneId API.
The java.util.TimeZone and java.time.ZoneId have inconsistent behavior.
cuDF have java.time compatible impl, but does not have java.util compatible impl
java.util.TimeZone.getOffset and java.time.ZoneId.getOffset are not always consistent.

For more details, click to expand

static void testDiffBehaviorBetweenTwoAPIs() {
  // diff in `Africa/Casablanca` timezone at 6424721300000: 0 vs 3600000
  String tzId = "Africa/Casablanca";
  long epochMillis = 6424721300000L;
  int offsetMillis_1 = java.util.TimeZone.getTimeZone(tzId).getOffset(epochMillis);
  int offsetMillis_2 = java.time.ZoneId.of(tzId, ZoneId.SHORT_IDS).getRules().getOffset(Instant.ofEpochMilli(epochMillis)).getTotalSeconds() * 1000;
  if (offsetMillis_1 != offsetMillis_2) {
    // print: get diff!! 0 vs 3600000
    System.out.println("get diff!! " + offsetMillis_1 + " vs " + offsetMillis_2);
  }
}

Solution 1 [not feasible], use cuDF with ignoreTimezoneInStripeFooter=False.

cuDF manages the ORC writer timezone decode.
Problem: for far-future timestamps projected into the synthetic 400-year cycle, dates before the first synthetic DST transition were incorrectly using the first cycle entry, which is the DST offset. That causes exactly the +1 hour winter drift you saw for America/Los_Angeles with years like 8770.
cuDF has java.time compatible impl instead of java.util. We can not get correct result.
So this solution is not feasible.

Solution 2, develope kernel, use cuDF with ignoreTimezoneInStripeFooter=True

cuDF does not manage the ORC writer timezone decode. Decode as UTC in cuDF.
All time rebasing logic is handled by customized JNI kernel which is compatible to java.util.

changes

• Remove the orc_timezone_info.data file, get the timezone info dynamically.
  Why: A pre-built timezone info file can not handle for all Java versions, in future the timezone info may change.
• Implements java.util.Timezone logic
• Add OrcTimezoneSuite to test reader/writer timezone combinations.

Related cuDF issue

• ORC reader returns incorrect timestamp for epoch boundary values rapidsai/cudf#21993

perf number

Test	CPU avg (ms)	GPU avg (ms)	Speedup
cross-tz (LA→UTC)	23,662	13,815	1.71x
same-tz-baseline (LA→LA)	79,699	13,766	5.79x

How to run, refer to `OrcTimezonePerfSuite.scala`, click to expand

argLine="-DenableOrcTimeZonePerf=true \
         -DorcPerfWriterTZ=America/Los_Angeles \
         -DorcPerfReaderTZ=UTC \
         -DorcPerfRows=1073741824" \
mvn test -Dbuildver=350 \
  -DwildcardSuites=com.nvidia.spark.rapids.timezone.OrcTimezonePerfSuite

Checklists

Documentation

• Updated for new or modified user-facing features or behaviors
• No user-facing change

Testing

• Added or modified tests to cover new code paths
• Covered by existing tests
  (Please provide the names of the existing tests in the PR description.)
• Not required

Performance

• Tests ran and results are added in the PR description
• Issue filed with a link in the PR description
• Not required

Signed-off-by: Chong Gao chongg@nvidia.com

[greptile-apps]

Greptile Summary

This PR adds proper DST-aware timezone conversion for ORC reads on GPU. It replaces the prior ignoreTimezoneInStripeFooter=false approach with a new JNI kernel (convertOrcTimezones) that matches java.util.TimeZone semantics (which ORC Java OSS uses), resolving long-standing drift for DST timezones.

• GpuOrcTimezoneUtils centralises timezone resolution (resolveWriterTimezone via ZoneId.SHORT_IDS) and the rebaseOrcTimestamps rebase logic; the read pipeline now tracks writerTimezone from each stripe footer and passes it down to both the chunked and non-chunked table producers.
• The coalescing reader is extended to split batches when files carry different writer timezone rules; the GPU ORC write path now falls back to CPU for non-UTC JVMs because cuDF's ORC writer always stamps writerTimezone="UTC" regardless of the JVM default.
• The old experimental spark.rapids.sql.orc.read.ignore.write.timezone config is removed, Python integration tests drop the @allow_non_gpu guards that existed for non-UTC sessions, and new OrcTimezoneSuite / OrcTimezonePerfSuite Scala tests cover the writer/reader matrix.

Confidence Score: 5/5

Safe to merge; the core timezone rebasing logic, resource lifecycle, and coalescing split checks are all correct.

The change is well-structured: timezone resolution uses ZoneId.of with SHORT_IDS (avoiding the silent-GMT fallback of TimeZone.getTimeZone), resource management in rebaseWithWriterTimezone uses withResource/safeMap correctly, the chunked and non-chunked table producer paths both call the rebase step, and the coalescing reader correctly splits batches on timezone-rule mismatches. The only finding is that the STRUCT branch in rebaseNestedWithWriterTimezone always allocates a new ColumnView even when no child changed — unlike the LIST branch which short-circuits — causing an unnecessary copyToColumnVector for STRUCT columns without timestamp descendants. This is a performance nit, not a data-correctness regression.

Files Needing Attention: GpuOrcTimezoneUtils.scala — the STRUCT short-circuit optimisation in rebaseNestedWithWriterTimezone.

Important Files Changed

Filename	Overview
sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuOrcTimezoneUtils.scala	New utility for ORC timezone rebasing; resource management looks correct but the STRUCT branch in rebaseNestedWithWriterTimezone always allocates a new ColumnView even when no children changed, causing an unnecessary copyToColumnVector in the caller.
sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuOrcScan.scala	Writer timezone threaded through the full read pipeline; ignoreTimezoneInStripeFooter set unconditionally; mixed-stripe detection and coalescing split check look correct.
sql-plugin/src/main/scala/org/apache/spark/sql/rapids/GpuOrcFileFormat.scala	Adds GPU→CPU fallback for non-UTC timestamp writes to prevent interoperability issues; rationale is well-documented.
sql-plugin/src/main/scala/com/nvidia/spark/rapids/RapidsConf.scala	Removes the experimental ORC_READ_IGNORE_WRITE_TIMEZONE internal config and its accessor; straightforward cleanup.
sql-plugin/src/main/scala/com/nvidia/spark/rapids/shims/OrcCastingShims.scala	Return type of castIntegerToTimestamp tightened from ColumnView to ColumnVector; backward compatible since ColumnVector extends ColumnView.
tests/src/test/scala/com/nvidia/spark/rapids/timezone/OrcTimezoneSuite.scala	Comprehensive writer/reader timezone matrix tests; uses fixed seed (42L) for random data, restores JVM default timezone in finally blocks, and verifies GPU execution with existClasses.
tests/src/test/scala/com/nvidia/spark/rapids/timezone/OrcTimezonePerfSuite.scala	Perf-gated by system property; uses UUID-suffixed temp path to avoid cross-run interference; FileUtils.deleteRecursively resolves correctly to the project-local test utility.
integration_tests/src/main/python/orc_test.py	Removes non_utc_allow_orc_scan guards; consistent with GPU now supporting non-UTC ORC reads. test_compress_read_round_trip never had timestamps so the removal is safe.

Sequence Diagram

sequenceDiagram
    participant S as ORC Stripe Footer
    participant B as buildOutputStripes
    participant P as MakeOrcTableProducer
    participant U as GpuOrcTimezoneUtils
    participant J as GpuTimeZoneDB (JNI)

    S->>B: getWriterTimezone() per stripe
    B->>B: resolveWriterTimezone(id) via ZoneId.SHORT_IDS
    B->>B: writerTimezonesShareRules(distinctTzs)
    B-->>P: writerTimezone: ZoneId
    P->>U: rebaseOrcTimestamps(table, writerTimezone)
    U->>J: buildOrcTimezoneContext(writerTz, readerTz)
    J-->>U: OrcTimezoneContext
    U->>J: convertOrcTimezones(col, tzCtx) per timestamp col
    J-->>U: rebased ColumnVector
    U-->>P: rebased Table (input closed)
    P->>P: evolveSchemaIfNeededAndClose(rebased, ...)

Loading

_{Reviews (11): Last reviewed commit: "Allow expected non-UTC ORC write fallbac..." | Re-trigger Greptile}

[greptile-apps]

Tip:

Greploops — Automatically fix all review issues by running /greploops in Claude Code. It iterates: fix, push, re-review, repeat until 5/5 confidence.

Use the Greptile plugin for Claude Code to query reviews, search comments, and manage custom context directly from your terminal.

[firestarman]

Can you address the comments from greptile first ?

[firestarman]

One low-priority nit from my pass.

[res-life]

build

[res-life]

This is a cross-implementation ORC interoperability issue, not data corruption in libcudf itself

Tracked in rapidsai/cudf#23422. I reproduced it end-to-end with a real cudf::io::write_orc file (via cudf 25.12) and the exact Apache ORC Java 1.9.1 reader that Spark 3.5.0 ships.

Reproduction

A TIMESTAMP_MICROSECONDS column [0, 1s, 2s] written by libcudf, read back with the ORC Java reader:

reader JVM TZ	epoch micros read back
UTC	0, 1000000, 2000000 ✅
Asia/Shanghai	-28800000000, -28799000000, -28798000000 ❌ (shifted by −8h)

Root cause

libcudf writes the column as a plain ORC timestamp with writerTimezone="UTC" and stores raw UTC-instant seconds (seconds - orc_utc_epoch; stripe_enc.cu / writer_impl.cu). But cudf/Spark timestamps are instants, whereas the plain ORC timestamp type has local-datetime semantics. For a plain timestamp, the ORC Java reader unconditionally applies writerOffset − readerOffset whenever writer/reader zone rules differ (TreeReaderFactory.readTimestamp → SerializationUtils.convertBetweenTimezones). So in a non-UTC JVM session it shifts every value by −(reader offset).

The round-trip contract of a plain timestamp is that writerTimezone matches the zone the stored values are anchored to. Spark's CPU ORC writer honors this by stamping writerTimezone = the session/JVM zone, so the reader's conversion cancels. libcudf instead hard-codes writerTimezone="UTC" while the consuming Spark reader is in a non-UTC session, so the conversion does not cancel.

Why the other paths are fine (round-trip safe)

write → read	result
libcudf → libcudf	✅ correct (reader compensates symmetrically via the file's writerTimezone; independent of process TZ)
libcudf → ORC C++ (liborc / pyarrow)	✅ correct (C++ reader does not apply the conversion)
libcudf → ORC Java (Spark), non-UTC session	❌ shifted
Spark CPU → ORC Java	✅ correct (writes writerTimezone = session TZ, so rules match, offset = 0)
pyarrow → any	✅ correct (writes timestamp with local time zone, an instant type that is never converted)

Direction

The libcudf ORC output is self-consistent and valid for the ORC C++ reader, but it is not round-trip-safe against the ORC Java reader that Spark/Hive use in a non-UTC session. Two ways to fix:

1. cudf-spark side (this PR): convert UTC timestamps to the JVM writer timezone before calling write_orc, so the Java reader's writerOffset − readerOffset cancels. Fastest, keeps the workaround local.
2. libcudf side ([FEA] ORC writer: allow setting writerTimezone instead of hardcoding "UTC" rapidsai/cudf#23422): emit timestamp with local time zone (instant type — proven to round-trip through the Java reader) or accept an explicit writerTimezone. This is the more complete fix.

[res-life]

build

[firestarman]

lgtm

[revans2]

This looks good as a patch for reading ORC timestamps, but I am really concerned about rapidsai/cudf#23422 and I am not sure that cudf takes the local into account when writing. From what the AI tells me it does not look at the local at all. So that means we are either handing the wrong things to CUDF when we write, or our read code is not interpreting what we handed to cudf correctly. I don't think this is a cudf bug, but I could be wrong. But it is a blocker for any release that we have. This could be the difference between a TIMESTAMP and a TIMESTAMP_INSTANT being written out to the file. I think we need to dig deeper before we say that cudf is wrong here.

[revans2]

Have you validated that this is indeed a cudf issue? To me this is a blocker P0 issue and reading/writing timestamps in ORC for anything but UTC should probably be disabled until we can fix this.

[res-life]

Confirmed it's a libcudf writer bug, and agreed it's a blocker.

Root cause: ORC's plain timestamp is timezone-agnostic — on read the value is shifted by convertBetweenTimezones(writerTimezone, readerTimezone), so correctness depends entirely on the writerTimezone stored in each StripeFooter. The Spark CPU writer stamps the actual JVM zone (e.g. Asia/Shanghai), so a same-zone read cancels out. libcudf hardcodes writerTimezone = "UTC" (cpp/src/io/orc/writer_impl.cu:2674) and exposes no writer-timezone option at any layer (C++ orc_writer_options, cuDF Java ORCWriterOptions, spark-rapids-jni). So a GPU file written in a non-UTC JVM carries writerTimezone="UTC" and is read back shifted by the zone offset (−8h for Asia/Shanghai, matching the repro). It's not a plugin data bug — we hand cudf the correct UTC instants; cudf just can't record the real writer zone.

On the TIMESTAMP vs TIMESTAMP_INSTANT point: exactly — the file is only correct under UTC-reader/instant semantics, but Spark reads it as a local timestamp. Fixing this needs two coupled changes inside libcudf (stamp the real zone and base the seconds on that zone's 2015 epoch); setting the footer field alone still reads wrong, which is why it can't be worked around via existing options. Tracked in rapidsai/cudf#23422.

For this PR: the read path is fixed here and covered by the timezone matrix; only GPU write in non-UTC is affected. I'll restore the write-side CPU fallback for non-UTC timestamp ORC writes (instead of the xfail) so we never emit CPU-incompatible files until the cudf fix lands.

[res-life]

build

[res-life]

build

[res-life]

build

[res-life]

build

[res-life]

@revans2 Dug into this as you asked. You're right on both counts — and the result changes the ask rather than the diagnosis.

we are either handing the wrong things to CUDF when we write, or our read code is not interpreting what we handed to cudf correctly

It's the first one, and the mechanism is narrower than my issue originally claimed. libcudf and the Spark CPU writer store byte-identical values. The only thing that differs is one metadata string in the stripe footer.

With writer zone A (offset a), reader zone B (offset b), true instant I:

writer	stored V	footer	ORC Java reads	vs CPU
Spark CPU	I	A	I + a - b	—
libcudf	I	"UTC"	I - b	-a

libcudf hardcodes writerTimezone = "UTC" at writer_impl.cu:2686, unconditionally, and orc_writer_options exposes no setter for it. So the whole error is -a, and it comes entirely from a field we cannot set.

This could be the difference between a TIMESTAMP and a TIMESTAMP_INSTANT being written out to the file.

Right axis — and it's exactly why I am not proposing we switch the type. Spark writes plain TIMESTAMP for TimestampType (OrcUtils.scala:379-382); only the newer TimestampLTZNanosType maps to TIMESTAMP_INSTANT. If libcudf emitted TIMESTAMP_INSTANT, GPU files would carry a different schema than CPU files for identical input. Semantically cleaner, worse for interop. Wrong lever.

I don't think this is a cudf bug

Agreed that cudf is not miscomputing anything — its own reader and liborc both round-trip its files correctly under any TZ. But that is also why it looked fine for so long: cudf's reader shares cudf's assumption. The divergence only shows up against the ORC Java reader, which is what Spark, Hive, and Trino actually use.

One more thing that came out of this, which I think settles it. Before this PR, the old path was gated by spark.rapids.sql.orc.read.ignore.write.timezone, and with it enabled the GPU reader ignored the footer entirely and returned V as-is. Running that end-to-end:

path	result	vs CPU (I + a - b)
CPU write → CPU read	I + a - b	—
GPU write → CPU read	I - b	-a
GPU write → GPU read	I	b - a

So GPU→GPU was only ever correct when A = B — and the old test set that config with the comment "tz_sensitive_test guarantees the write and read are in the same timezone," which is precisely the one condition that makes b - a vanish. The two errors (writer lying in the footer, reader ignoring the footer) cancelled inside the GPU closed loop and nowhere else. That config is removed in this PR.

So where this leaves us:

• Read path — fixed properly here. It now resolves writerTimezone from the footer and reproduces java.util.TimeZone semantics for arbitrary writer/reader zone pairs, including alias IDs (PST, US/Pacific), DST boundaries, and the far-future cases. A != B is a real tested dimension for the first time.
• Write path — cannot be fixed the same way from our side. The only caller-side workaround is pre-shifting values to I + a under a footer still claiming UTC, which is wrong values and wrong metadata, correct only for readers in A. So this PR falls back to CPU for non-UTC timestamp writes instead.

That fallback is a pure perf loss caused by one unsettable field, which is the concrete cost to point at. I've rewritten rapidsai/cudf#23422 accordingly — retitled to [FEA] ORC writer: allow setting writerTimezone instead of hardcoding "UTC", asking for a setter that defaults to "UTC" so existing behavior is unchanged. That makes our output byte-identical to the CPU writer, and we can drop the fallback.

Does that address the blocker, or would you rather hold the PR until the cudf side lands?

[firestarman]

I am ok with the fallback of the ORC write with Non-UTC timezones now.