Skip to content

Commit 08289e8

Browse files
committed
fix: validate T-vs-space in incompatible path instead of CORRECTED_COMPATIBLE_FORMATS
Adding the format to CORRECTED_COMPATIBLE_FORMATS made it always go to GPU, but cuDF's %3f requires exactly 3 fractional digits while CPU accepts 1-3. This caused ParseDateTimeSuite to fail for inputs like "1999-12-31 11:59:59.9". Instead, keep the format on the incompatible path (so it falls back to CPU without incompatDateFormats) and add targeted validation there: - Reject inputs with T where the format expects space (fixes #13759) - Use cuDF isTimestamp to reject truly unparseable strings Signed-off-by: Allen Xu <allxu@nvidia.com> Made-with: Cursor
1 parent 2afff86 commit 08289e8

1 file changed

Lines changed: 20 additions & 6 deletions

File tree

sql-plugin/src/main/scala/org/apache/spark/sql/rapids/datetimeExpressions.scala

Lines changed: 20 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -607,8 +607,6 @@ object GpuToTimestamp {
607607
raw"\A\d{2}/\d{2}/\d{4}\Z"),
608608
"yyyy-MM-dd HH:mm:ss" -> ParseFormatMeta(Option('-'), isTimestamp = true,
609609
raw"\A\d{4}-\d{2}-\d{2}[ T]\d{2}:\d{2}:\d{2}\Z"),
610-
"yyyy-MM-dd HH:mm:ss.SSS" -> ParseFormatMeta(Option('-'), isTimestamp = true,
611-
raw"\A\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}\.\d{3}\Z"),
612610
"MM-dd" -> ParseFormatMeta(Option('-'), isTimestamp = false,
613611
raw"\A\d{2}-\d{2}\Z"),
614612
"MM/dd" -> ParseFormatMeta(Option('/'), isTimestamp = false,
@@ -708,10 +706,26 @@ object GpuToTimestamp {
708706
}
709707
}
710708
case _ =>
711-
// this is the incompatibleDateFormats case where we do not guarantee compatibility with
712-
// Spark and assume that all non-null inputs are valid
713-
withResource(Scalar.fromBool(true)) { s =>
714-
ColumnVector.fromScalar(s, col.getRowCount.toInt)
709+
// This is the incompatibleDateFormats case where we do not guarantee full
710+
// compatibility with Spark. However, we still reject inputs where the
711+
// date-time separator doesn't match the format: cuDF treats 'T' and space
712+
// as interchangeable, but Spark's DateTimeFormatter requires an exact match.
713+
// We also use cuDF isTimestamp to reject truly unparseable strings.
714+
if (strfFormat.startsWith("%Y-%m-%d %H") ||
715+
strfFormat.startsWith("%Y/%m/%d %H")) {
716+
val tProg = new RegexProgram(
717+
raw"\A.{10}T", CaptureGroups.NON_CAPTURE)
718+
withResource(col.matchesRe(tProg)) { hasT =>
719+
withResource(hasT.not()) { noT =>
720+
withResource(col.isTimestamp(strfFormat)) { cudfValid =>
721+
noT.and(cudfValid)
722+
}
723+
}
724+
}
725+
} else {
726+
withResource(Scalar.fromBool(true)) { s =>
727+
ColumnVector.fromScalar(s, col.getRowCount.toInt)
728+
}
715729
}
716730
}
717731
}

0 commit comments

Comments
 (0)