Skip to content

Commit b246d7f

Browse files
stevomitricIsaac
andcommitted
[SPARK-57462][PYTHON][SQL] Drop now-dead nanos precision branch in assertSchemaEqual
Since typeName() now embeds the precision (e.g. timestamp_ntz(9)), compare_datatypes_ignore_nullable already distinguishes precisions at the typeName equality check, so the AnyTimestampNanoType branch (and its now-inaccurate comment claiming the type name does not distinguish precisions) is dead. Remove the branch and its unused import; behavior is unchanged -- same precision stays equal under ignoreNullable, and different precisions / micros-vs-nanos stay distinct. Co-authored-by: Isaac <no-reply@databricks.com>
1 parent ebd7a72 commit b246d7f

1 file changed

Lines changed: 1 addition & 6 deletions

File tree

python/pyspark/testing/utils.py

Lines changed: 1 addition & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -41,7 +41,7 @@
4141
from pyspark.sql import Row
4242
from pyspark.sql.dataframe import DataFrame
4343
from pyspark.sql.functions import col, when
44-
from pyspark.sql.types import AnyTimestampNanoType, StructField, StructType, VariantVal
44+
from pyspark.sql.types import StructField, StructType, VariantVal
4545

4646
__all__ = ["assertDataFrameEqual", "assertSchemaEqual"]
4747

@@ -701,11 +701,6 @@ def compare_datatypes_ignore_nullable(dt1: Any, dt2: Any):
701701
elif dt1.typeName() == "decimal":
702702
# Fix for SPARK-51062: Compare precision and scale for decimal types
703703
return dt1.precision == dt2.precision and dt1.scale == dt2.scale
704-
elif isinstance(dt1, AnyTimestampNanoType):
705-
# SPARK-57462: the nanosecond timestamp types carry a fractional-second
706-
# precision, like decimal above; the type name alone does not distinguish
707-
# timestamp_ntz(7) from timestamp_ntz(9), so compare the precision too.
708-
return dt1.precision == dt2.precision
709704
elif dt1.typeName() == "struct":
710705
return compare_schemas_ignore_nullable(dt1, dt2)
711706
else:

0 commit comments

Comments
 (0)