Skip to content

[SPARK-59433][SQL] Render TIMESTAMP_LTZ explicitly when spark.sql.timestampType=TIMESTAMP_NTZ #58734

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
pan3793 wants to merge 1 commit into
base: master
Choose a base branch
from
Open

[SPARK-59433][SQL] Render TIMESTAMP_LTZ explicitly when spark.sql.timestampType=TIMESTAMP_NTZ #58734

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions sql/api/src/main/scala/org/apache/spark/sql/types/TimestampType.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@
package org.apache.spark.sql.types

import org.apache.spark.annotation.Stable
import org.apache.spark.sql.internal.SqlApiConf

/**
* The timestamp type represents a time instant in microsecond precision. Valid range is
Expand All @@ -35,6 +36,11 @@ class TimestampType private () extends DatetimeType {
*/
override def defaultSize: Int = 8

// A bare `TIMESTAMP` resolves to the session default timestamp type, so render this type
// explicitly when that default is not this type, keeping the generated DDL round-trippable.
override def sql: String =

@dongjoon-hyun dongjoon-hyun Sep 11, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This changes auto-generated column names too, not only DDL. Cast.sql and Literal.sql use dataType.sql, so under spark.sql.timestampType=TIMESTAMP_NTZ, SELECT CAST(c AS TIMESTAMP_LTZ) is now named CAST(c AS TIMESTAMP_LTZ) instead of CAST(c AS TIMESTAMP).

This can break existing persistent views after upgrade. A view created in an NTZ session, e.g. CREATE VIEW v AS SELECT CAST(ntz_col AS TIMESTAMP_LTZ) FROM t, stores the old query output column name and captures spark.sql.timestampType=TIMESTAMP_NTZ. On read, the view query is re-analyzed under the captured conf, produces the new name, and GetViewColumnByNameAndOrdinal fails with INCOMPATIBLE_VIEW_SCHEMA_CHANGE.

CTAS / DataFrame column names and toSQLType error messages change in the same way. Could you check this case and mention it in the user-facing change section?

if (SqlApiConf.get.timestampType == TimestampNTZType) "TIMESTAMP_LTZ" else "TIMESTAMP"

@gengliangwang gengliangwang Sep 11, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] Do not use client-local config to render LTZ. SqlApiConf is explicitly hardcoded in sql-api-only/Spark Connect, so this branch is never taken by a Scala Connect client even when the remote session has spark.sql.timestampType=TIMESTAMP_NTZ. For example, DataFrameReader sends a concrete StructType as schema.toDDL; the server reparses that DDL under the remote NTZ session, so a requested TimestampType is sent as bare TIMESTAMP and silently becomes TimestampNTZType. The concrete-schema overloads of from_json, from_csv, and from_xml have the same issue. Please use an unambiguous schema transport/rendering for concrete types and add Scala Connect coverage under an NTZ remote session.


override def equals(obj: Any): Boolean = obj.isInstanceOf[TimestampType]

override def hashCode(): Int = classOf[TimestampType].getSimpleName.hashCode
Expand Down
2 changes: 1 addition & 1 deletion sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/literals.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -691,7 +691,7 @@ case class Literal (value: Any, dataType: DataType) extends LeafExpression {
case (_: Long, _: TimeType) =>
s"TIME '$toString'"
case (v: Long, TimestampType) =>
s"TIMESTAMP '$toString'"
s"${dataType.sql} '$toString'"
case (v: Long, TimestampNTZType) =>
s"TIMESTAMP_NTZ '$toString'"
// toString strips trailing zeros (display-only); pad back to exactly `precision` digits
Expand Down
12 changes: 12 additions & 0 deletions ...yst/src/test/scala/org/apache/spark/sql/catalyst/expressions/LiteralExpressionSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -528,6 +528,18 @@ class LiteralExpressionSuite extends SparkFunSuite with ExpressionEvalHelper {
}
}

test("SPARK-59433: format timestamp literal respects the default timestamp type") {
withTimeZones(sessionTimeZone = "GMT+01:00", systemTimeZone = "GMT-08:00") {
val timestamp = LocalDateTime.of(2019, 3, 21, 0, 2, 3, 456000000)
.atZone(ZoneOffset.UTC)
.toInstant
assert(Literal.create(timestamp).sql === "TIMESTAMP '2019-03-21 01:02:03.456'")
withSQLConf(SQLConf.TIMESTAMP_TYPE.key -> SQLConf.TimestampTypes.TIMESTAMP_NTZ.toString) {
assert(Literal.create(timestamp).sql === "TIMESTAMP_LTZ '2019-03-21 01:02:03.456'")
}
}
}

test("format date literal independently from time zone") {
withTimeZones(sessionTimeZone = "GMT-11:00", systemTimeZone = "GMT-10:00") {
val date = LocalDate.of(2019, 3, 21)
Expand Down
215 changes: 215 additions & 0 deletions sql/core/src/test/resources/sql-tests/analyzer-results/timestampNTZ/timestamp-ltz.sql.out
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,215 @@
-- Automatically generated by SQLQueryTestSuite
-- !query
select timestamp_ltz'2016-12-31 00:12:00', timestamp_ltz'2016-12-31'
-- !query analysis
[Analyzer test output redacted due to nondeterminism]


-- !query
select to_timestamp_ltz(null), to_timestamp_ltz('2016-12-31 00:12:00'), to_timestamp_ltz('2016-12-31', 'yyyy-MM-dd')
-- !query analysis
Project [to_timestamp_ltz(cast(null as string), None, TimestampType, Some(America/Los_Angeles), true) AS to_timestamp_ltz(NULL)#x, to_timestamp_ltz(2016-12-31 00:12:00, None, TimestampType, Some(America/Los_Angeles), true) AS to_timestamp_ltz(2016-12-31 00:12:00)#x, to_timestamp_ltz(2016-12-31, Some(yyyy-MM-dd), TimestampType, Some(America/Los_Angeles), true) AS to_timestamp_ltz(2016-12-31, yyyy-MM-dd)#x]
+- OneRowRelation


-- !query
select to_timestamp_ltz(to_date(null)), to_timestamp_ltz(to_date('2016-12-31'))
-- !query analysis
Project [to_timestamp_ltz(to_date(cast(null as string), None, Some(America/Los_Angeles), true), None, TimestampType, Some(America/Los_Angeles), true) AS to_timestamp_ltz(to_date(NULL))#x, to_timestamp_ltz(to_date(2016-12-31, None, Some(America/Los_Angeles), true), None, TimestampType, Some(America/Los_Angeles), true) AS to_timestamp_ltz(to_date(2016-12-31))#x]
+- OneRowRelation


-- !query
select to_timestamp_ltz(to_timestamp_ntz(null)), to_timestamp_ltz(to_timestamp_ntz('2016-12-31 00:12:00'))
-- !query analysis
Project [to_timestamp_ltz(to_timestamp_ntz(cast(null as string), None, TimestampNTZType, Some(America/Los_Angeles), true), None, TimestampType, Some(America/Los_Angeles), true) AS to_timestamp_ltz(to_timestamp_ntz(NULL))#x, to_timestamp_ltz(to_timestamp_ntz(2016-12-31 00:12:00, None, TimestampNTZType, Some(America/Los_Angeles), true), None, TimestampType, Some(America/Los_Angeles), true) AS to_timestamp_ltz(to_timestamp_ntz(2016-12-31 00:12:00))#x]
+- OneRowRelation


-- !query
SELECT make_timestamp_ltz(2021, 07, 11, 6, 30, 45.678)
-- !query analysis
Project [make_timestamp_ltz(2021, 7, 11, 6, 30, cast(45.678 as decimal(16,6)), None, Some(America/Los_Angeles), true, TimestampType) AS make_timestamp_ltz(2021, 7, 11, 6, 30, 45.678)#x]
+- OneRowRelation


-- !query
SELECT make_timestamp_ltz(2021, 07, 11, 6, 30, 45.678, 'CET')
-- !query analysis
Project [make_timestamp_ltz(2021, 7, 11, 6, 30, cast(45.678 as decimal(16,6)), Some(CET), Some(America/Los_Angeles), true, TimestampType) AS make_timestamp_ltz(2021, 7, 11, 6, 30, 45.678, CET)#x]
+- OneRowRelation


-- !query
SELECT make_timestamp_ltz(2021, 07, 11, 6, 30, 60.007)
-- !query analysis
Project [make_timestamp_ltz(2021, 7, 11, 6, 30, cast(60.007 as decimal(16,6)), None, Some(America/Los_Angeles), true, TimestampType) AS make_timestamp_ltz(2021, 7, 11, 6, 30, 60.007)#x]
+- OneRowRelation


-- !query
SELECT make_timestamp_ltz(make_date(2021, 07, 11), make_time(6, 30, 45.678))
-- !query analysis
Project [make_timestamp(make_date(2021, 7, 11, true), Some(make_time(6, 30, cast(45.678 as decimal(16,6)))), None, Some(America/Los_Angeles)) AS make_timestamp(make_date(2021, 7, 11), make_time(6, 30, 45.678))#x]
+- OneRowRelation


-- !query
SELECT make_timestamp_ltz(NULL, TIME'00:00:00')
-- !query analysis
[Analyzer test output redacted due to nondeterminism]


-- !query
SELECT make_timestamp_ltz(DATE'1970-01-01', NULL)
-- !query analysis
[Analyzer test output redacted due to nondeterminism]


-- !query
SELECT make_timestamp_ltz(timestamp_ntz'2018-11-17 13:33:33', TIME'0:0:0')
-- !query analysis
org.apache.spark.sql.catalyst.ExtendedAnalysisException
{
"errorClass" : "DATATYPE_MISMATCH.UNEXPECTED_INPUT_TYPE",
"sqlState" : "42K09",
"messageParameters" : {
"inputSql" : "\"TIMESTAMP_NTZ '2018-11-17 13:33:33'\"",
"inputType" : "\"TIMESTAMP_NTZ\"",
"paramIndex" : "first",
"requiredType" : "\"DATE\"",
"sqlExpr" : "\"make_timestamp(TIMESTAMP_NTZ '2018-11-17 13:33:33', TIME '00:00:00')\""
},
"queryContext" : [ {
"objectType" : "",
"objectName" : "",
"startIndex" : 8,
"stopIndex" : 74,
"fragment" : "make_timestamp_ltz(timestamp_ntz'2018-11-17 13:33:33', TIME'0:0:0')"
} ]
}


-- !query
SELECT make_timestamp_ltz(DATE'2025-06-20', timestamp_ntz'2018-11-17 13:33:33')
-- !query analysis
org.apache.spark.sql.catalyst.ExtendedAnalysisException
{
"errorClass" : "DATATYPE_MISMATCH.UNEXPECTED_INPUT_TYPE",
"sqlState" : "42K09",
"messageParameters" : {
"inputSql" : "\"TIMESTAMP_NTZ '2018-11-17 13:33:33'\"",
"inputType" : "\"TIMESTAMP_NTZ\"",
"paramIndex" : "second",
"requiredType" : "\"TIME\"",
"sqlExpr" : "\"make_timestamp(DATE '2025-06-20', TIMESTAMP_NTZ '2018-11-17 13:33:33')\""
},
"queryContext" : [ {
"objectType" : "",
"objectName" : "",
"startIndex" : 8,
"stopIndex" : 79,
"fragment" : "make_timestamp_ltz(DATE'2025-06-20', timestamp_ntz'2018-11-17 13:33:33')"
} ]
}


-- !query
SELECT make_timestamp_ltz(make_date(2021, 07, 11), make_time(6, 30, 45.678), 'PST')
-- !query analysis
Project [make_timestamp(make_date(2021, 7, 11, true), Some(make_time(6, 30, cast(45.678 as decimal(16,6)))), Some(PST), Some(America/Los_Angeles)) AS make_timestamp(make_date(2021, 7, 11), make_time(6, 30, 45.678), PST)#x]
+- OneRowRelation


-- !query
SELECT make_timestamp_ltz(make_date(2021, 07, 11), make_time(6, 30, 45.678), 'CET')
-- !query analysis
Project [make_timestamp(make_date(2021, 7, 11, true), Some(make_time(6, 30, cast(45.678 as decimal(16,6)))), Some(CET), Some(America/Los_Angeles)) AS make_timestamp(make_date(2021, 7, 11), make_time(6, 30, 45.678), CET)#x]
+- OneRowRelation


-- !query
SELECT convert_timezone('Europe/Brussels', timestamp_ltz'2022-03-23 00:00:00 America/Los_Angeles')
-- !query analysis
[Analyzer test output redacted due to nondeterminism]


-- !query
SELECT try_make_timestamp_ltz(make_date(2021, 07, 11), make_time(6, 30, 45.678))
-- !query analysis
Project [make_timestamp(make_date(2021, 7, 11, true), Some(make_time(6, 30, cast(45.678 as decimal(16,6)))), None, Some(America/Los_Angeles)) AS make_timestamp(make_date(2021, 7, 11), make_time(6, 30, 45.678))#x]
+- OneRowRelation


-- !query
SELECT try_make_timestamp_ltz(NULL, TIME'00:00:00')
-- !query analysis
[Analyzer test output redacted due to nondeterminism]


-- !query
SELECT try_make_timestamp_ltz(DATE'1970-01-01', NULL)
-- !query analysis
[Analyzer test output redacted due to nondeterminism]


-- !query
SELECT try_make_timestamp_ltz(timestamp_ntz'2018-11-17 13:33:33', TIME'0:0:0')
-- !query analysis
org.apache.spark.sql.catalyst.ExtendedAnalysisException
{
"errorClass" : "DATATYPE_MISMATCH.UNEXPECTED_INPUT_TYPE",
"sqlState" : "42K09",
"messageParameters" : {
"inputSql" : "\"TIMESTAMP_NTZ '2018-11-17 13:33:33'\"",
"inputType" : "\"TIMESTAMP_NTZ\"",
"paramIndex" : "first",
"requiredType" : "\"DATE\"",
"sqlExpr" : "\"make_timestamp(TIMESTAMP_NTZ '2018-11-17 13:33:33', TIME '00:00:00')\""
},
"queryContext" : [ {
"objectType" : "",
"objectName" : "",
"startIndex" : 8,
"stopIndex" : 78,
"fragment" : "try_make_timestamp_ltz(timestamp_ntz'2018-11-17 13:33:33', TIME'0:0:0')"
} ]
}


-- !query
SELECT try_make_timestamp_ltz(DATE'2025-06-20', timestamp_ntz'2018-11-17 13:33:33')
-- !query analysis
org.apache.spark.sql.catalyst.ExtendedAnalysisException
{
"errorClass" : "DATATYPE_MISMATCH.UNEXPECTED_INPUT_TYPE",
"sqlState" : "42K09",
"messageParameters" : {
"inputSql" : "\"TIMESTAMP_NTZ '2018-11-17 13:33:33'\"",
"inputType" : "\"TIMESTAMP_NTZ\"",
"paramIndex" : "second",
"requiredType" : "\"TIME\"",
"sqlExpr" : "\"make_timestamp(DATE '2025-06-20', TIMESTAMP_NTZ '2018-11-17 13:33:33')\""
},
"queryContext" : [ {
"objectType" : "",
"objectName" : "",
"startIndex" : 8,
"stopIndex" : 83,
"fragment" : "try_make_timestamp_ltz(DATE'2025-06-20', timestamp_ntz'2018-11-17 13:33:33')"
} ]
}


-- !query
SELECT try_make_timestamp_ltz(make_date(2021, 07, 11), make_time(6, 30, 45.678), 'PST')
-- !query analysis
Project [make_timestamp(make_date(2021, 7, 11, true), Some(make_time(6, 30, cast(45.678 as decimal(16,6)))), Some(PST), Some(America/Los_Angeles)) AS make_timestamp(make_date(2021, 7, 11), make_time(6, 30, 45.678), PST)#x]
+- OneRowRelation


-- !query
SELECT try_make_timestamp_ltz(make_date(2021, 07, 11), make_time(6, 30, 45.678), 'CET')
-- !query analysis
Project [make_timestamp(make_date(2021, 7, 11, true), Some(make_time(6, 30, cast(45.678 as decimal(16,6)))), Some(CET), Some(America/Los_Angeles)) AS make_timestamp(make_date(2021, 7, 11), make_time(6, 30, 45.678), CET)#x]
+- OneRowRelation
Loading