[FEA] ORC writer: allow setting writerTimezone instead of hardcoding "UTC"

[res-life]

Summary

libcudf's ORC writer never records the actual writer timezone — it hardcodes writerTimezone="UTC" at cpp/src/io/orc/writer_impl.cu:2686 with no writer-side option. The file therefore misdeclares its own timezone, and any spec-conforming ORC reader in a non-UTC zone is guaranteed to read it shifted.

Note this is not a missing field. The field is present and set to "UTC", which is precisely why Apache ORC Java actively applies an off(UTC) - off(reader) correction rather than skipping conversion.

The asymmetry

The reader options already expose timezone control; the writer options expose none.

	timezone control
orc_reader_options	✅ ignore_timezone_in_stripe_footer (cpp/include/cudf/io/orc.hpp)
orc_writer_options	❌ nothing

That gap is the entire issue. Everything below follows from it.

Why it breaks interop

ORC's plain timestamp is a wall-clock type; readers apply off(writerTZ) - off(readerTZ). Hive/Spark carry instants through it by tagging the footer with the writing JVM's zone so the correction cancels. libcudf stores the identical values but cannot tag anything but UTC.

Writer zone A (offset a), reader zone B (offset b), true instant I:

writer	stored V	footer	ORC Java reads	vs Spark CPU
Spark CPU	I	A	I + a - b	—
libcudf	I	"UTC"	I - b	-a

The stored values are byte-identical. Only the footer differs, and the entire error is -a.

Measured with A = B = Asia/Shanghai (a = b = 8h), input epoch 0s, 1s, 2s:

CPU-written,     read by Spark CPU:            0        1000000        2000000
libcudf-written, read by Spark CPU:  -28800000000   -28799000000   -28798000000

Exactly -a. Full C++ repro and Java reader harness in #23422 (comment).

Testing with C++ tooling alone will not reproduce this — liborc and cudf's own reader share cudf's assumption. The divergence only appears with the ORC Java reader, which is what Spark, Hive, and Trino use.

Requested fix

Add a writerTimezone setter to orc_writer_options / its builder, defaulting to the current "UTC" so existing behavior is unchanged.

That makes libcudf output byte-identical to the Spark CPU writer. The two alternatives are both worse:

• Emit TIMESTAMP_INSTANT instead. Semantically cleaner, but Spark writes plain TIMESTAMP for TimestampType (OrcUtils.scala:379-382), so GPU files would carry a different schema than CPU files for identical input.
• Pre-shift values caller-side. Produces V = I + a under a footer still claiming UTC — wrong values and wrong metadata, correct only for readers in A.

Current cost

NVIDIA/cudf-spark#14544 has to disable GPU ORC timestamp writes entirely in any non-UTC JVM and fall back to CPU, purely because of this one unsettable field:

if (types.exists(GpuOverrides.isOrContainsTimestamp) &&
    !GpuOverrides.isUTCTimezone(ZoneId.systemDefault())) {
  meta.willNotWorkOnGpu("Writing ORC timestamps is only supported in the UTC timezone ...")
}

The read path in that PR was fixed properly — it now reproduces java.util.TimeZone semantics for arbitrary writer/reader zone pairs. The write path cannot be fixed the same way from the caller side.

Environment

cudf 25.12 · Apache ORC Java 1.9.1 (Spark 3.5.0) · JDK 17

[res-life]

Root cause (confirmed) and a self-contained C++ reproduction

This is a cross-implementation interoperability problem, and it is important to note up front: the libcudf-written file is read correctly by the Apache ORC C++ reader (liborc); the shift only appears with the Apache ORC Java reader (which is what Spark/Hive use). So a maintainer testing with C++ tooling alone will not see the bug — you must read with the Java reader.

What libcudf writes

For a TIMESTAMP_* column the ORC writer emits:

• a plain ORC timestamp column (local-datetime semantics), not timestamp with local time zone (instant semantics),
• writerTimezone = "UTC" (cpp/src/io/orc/writer_impl.cu),
• stored seconds = utc_seconds - orc_utc_epoch where orc_utc_epoch = 1420070400 (cpp/src/io/orc/orc.hpp, cpp/src/io/orc/stripe_enc.cu).

Because cudf timestamps are instants but the plain ORC timestamp type is a local datetime, the ORC Java reader applies writerOffset − readerOffset whenever writer/reader zone rules differ (TreeReaderFactory.readTimestamp → SerializationUtils.convertBetweenTimezones). In a non-UTC JVM session every value shifts by −(reader offset).

Step 1 — write the file with libcudf (C++)

// repro_orc_tz.cpp
// Build in a libcudf dev env, e.g.:
//   g++ repro_orc_tz.cpp -std=c++17 $(pkg-config --cflags --libs cudf) -lcudart -o repro
#include <cudf/column/column_factories.hpp>
#include <cudf/io/orc.hpp>
#include <cudf/table/table_view.hpp>
#include <cudf/types.hpp>
#include <cuda_runtime.h>
#include <cstdint>
#include <vector>

int main() {
  // TIMESTAMP_MICROSECONDS column: epoch 0s, 1s, 2s (pure UTC instants)
  std::vector<int64_t> host{0, 1'000'000, 2'000'000};

  auto col = cudf::make_fixed_width_column(
      cudf::data_type{cudf::type_id::TIMESTAMP_MICROSECONDS},
      host.size(), cudf::mask_state::UNALLOCATED);
  cudaMemcpy(col->mutable_view().data<int64_t>(), host.data(),
             host.size() * sizeof(int64_t), cudaMemcpyHostToDevice);

  auto tbl = cudf::table_view{{col->view()}};
  cudf::io::table_input_metadata meta(tbl);
  meta.column_metadata[0].set_name("ts");

  auto opts = cudf::io::orc_writer_options::builder(
                  cudf::io::sink_info{"/tmp/cudf_cpp.orc"}, tbl)
                  .metadata(std::move(meta));
  cudf::io::write_orc(opts);   // -> writerTimezone="UTC", plain `timestamp`, seconds-1420070400
  return 0;
}

The Python one-liner cudf.DataFrame({'ts': cudf.Series(pd.to_datetime([0,1,2], unit='s'))}).to_orc('/tmp/cudf_cpp.orc') goes through the same cudf::io::write_orc path and produces byte-identical output.

Step 2 — read it back with the Apache ORC Java reader (what Spark uses)

// OrcRead.java  — classpath: orc-core (shaded), orc-shims, hadoop-client-api/runtime,
//                 hive-storage-api, aircompressor   (e.g. from a Spark 3.5 jars/ dir)
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.orc.*;
import org.apache.hadoop.hive.ql.exec.vector.*;

public class OrcRead {
  public static void main(String[] a) throws Exception {
    Reader r = OrcFile.createReader(new Path(a[0]), OrcFile.readerOptions(new Configuration()));
    RecordReader rows = r.rows();
    VectorizedRowBatch b = r.getSchema().createRowBatch();
    while (rows.nextBatch(b)) {
      TimestampColumnVector ts = (TimestampColumnVector) b.cols[0];
      for (int i = 0; i < b.size; i++)
        System.out.println(ts.time[i] / 1000 * 1_000_000L + ts.nanos[i] / 1000L);
    }
    rows.close();
  }
}

$ TZ=UTC           java -cp "<spark>/jars/*:." OrcRead /tmp/cudf_cpp.orc
0
1000000
2000000
$ TZ=Asia/Shanghai java -cp "<spark>/jars/*:." OrcRead /tmp/cudf_cpp.orc
-28800000000
-28799000000
-28798000000          # <-- shifted by the Asia/Shanghai UTC offset (-8h)

Control results (proving it is a Java-vs-C++ divergence, not corruption)

• liborc / pyarrow (ORC C++) reader on the same file: 0, 1000000, 2000000 under any TZ — no shift.
• libcudf reader on the same file: correct under any TZ (the cudf reader compensates symmetrically via the file's writerTimezone and does not depend on the process TZ).
• A file written as timestamp with local time zone (e.g. by pyarrow) reads back correctly through the Java reader under Asia/Shanghai.
• A plain timestamp file whose writerTimezone equals the reader zone (what Spark's CPU ORC writer emits) also round-trips, because the writerOffset − readerOffset term cancels.

Fix directions

1. Emit timestamp with local time zone (instant type) for instant timestamp data — proven to round-trip through the Java reader.
2. Or allow the caller to set writerTimezone so it can match the consumer's session zone (the workaround used on the cudf-spark side).

Environment: cudf 25.12, Apache ORC Java 1.9.1 (the version shipped with Spark 3.5.0), JDK 17.