Skip to content

Commit 8e360c1

Browse files
authored
Handle collect_set signed zeros by Scala version [databricks] (#15322)
Fixes #12199. ### Description Spark CPU window `collect_set` behavior for `0.0` and `-0.0` depends on the Scala binary version. Spark builds using Scala 2.12 can retain both signed zeros in some window frames, while Scala 2.13 builds and the GPU implementation deduplicate them. This change excludes `-0.0` from the float special cases only within `test_window_aggs_for_rows_collect_set` when it runs against Scala 2.12. The shared collect-set generator is unchanged, so other tests retain their existing coverage. Scala 2.13 continues to exercise both signed-zero values. No runtime behavior changes. Testing: - `python3 -m py_compile integration_tests/src/main/python/window_function_test.py` - Spark 3.5.9 / Scala 2.12 CPU A/B validation with `DATAGEN_SEED=1740159968`: the original generator produced 71 signed-zero duplicate window frames out of 2,048 rows; the PR-localized generator produced 0 - Spark 3.5.6 / Scala 2.12 GPU integration test with `DATAGEN_SEED=1740159968` and `TZ=America/Punta_Arenas`: 1 passed - Spark 4.0.1 / Scala 2.13 package build: success - Spark 4.0.1 / Scala 2.13 GPU integration test with the same seed and time zone: 1 passed - Other window collect-set candidates using the shared generator on Spark 3.5.6 / Scala 2.12: 3 passed - Representative hash groupby/reduction collect-set suites on Spark 3.5.6 / Scala 2.12: 42 passed ### Checklists Documentation - [ ] Updated for new or modified user-facing features or behaviors - [x] No user-facing change Testing - [x] Added or modified tests to cover new code paths - [ ] Covered by existing tests (Please provide the names of the existing tests in the PR description.) - [ ] Not required Performance - [ ] Tests ran and results are added in the PR description - [ ] Issue filed with a link in the PR description - [x] Not required --------- Signed-off-by: Chong Gao <chongg@nvidia.com>
1 parent 4383de9 commit 8e360c1

1 file changed

Lines changed: 19 additions & 2 deletions

File tree

integration_tests/src/main/python/window_function_test.py

Lines changed: 19 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -23,7 +23,7 @@
2323
import pyspark.sql.functions as f
2424
from spark_session import is_before_spark_320, is_databricks113_or_later, \
2525
is_databricks133_or_later, is_spark_350_or_later, spark_version, with_cpu_session, \
26-
is_spark_340_or_later, is_spark_420_or_later
26+
is_scala212, is_spark_340_or_later, is_spark_420_or_later
2727
import warnings
2828

2929
# mark this test as ci_1 for mvn verify sanity check in pre-merge CI
@@ -2377,8 +2377,25 @@ def do_it(spark):
23772377
@ignore_order(local=True)
23782378
@allow_non_gpu(*non_utc_allow)
23792379
def test_window_aggs_for_rows_collect_set():
2380+
data_gen = _gen_data_for_collect_set
2381+
if is_scala212():
2382+
# Scala 2.12 CPU window collect_set can retain both signed zeros, while the GPU and
2383+
# Scala 2.13 treat them as the same value. Exclude -0.0 from this Scala 2.12 test.
2384+
float_special_cases = [
2385+
FLOAT_MIN, FLOAT_MAX, 0.0, 1.0, -1.0,
2386+
float('inf'), float('-inf'), float('nan'), NEG_FLOAT_NAN_MAX_VALUE]
2387+
double_special_cases = [
2388+
DOUBLE_MIN, DOUBLE_MAX, 0.0, 1.0, -1.0,
2389+
float('inf'), float('-inf'), float('nan'), NEG_DOUBLE_NAN_MAX_VALUE]
2390+
collect_set_fp_gens = {
2391+
'c_float': RepeatSeqGen(FloatGen(special_cases=float_special_cases), length=15),
2392+
'c_double': RepeatSeqGen(DoubleGen(special_cases=double_special_cases), length=15)}
2393+
data_gen = [
2394+
(name, collect_set_fp_gens[name]) if name in collect_set_fp_gens else (name, gen)
2395+
for name, gen in data_gen]
2396+
23802397
assert_gpu_and_cpu_are_equal_sql(
2381-
lambda spark: gen_df(spark, _gen_data_for_collect_set),
2398+
lambda spark: gen_df(spark, data_gen),
23822399
"window_collect_table",
23832400
'''
23842401
select a, b,

0 commit comments

Comments
 (0)