Commit 8e360c1
authored
Handle collect_set signed zeros by Scala version [databricks] (#15322)
Fixes #12199.
### Description
Spark CPU window `collect_set` behavior for `0.0` and `-0.0` depends on
the Scala binary version. Spark builds using Scala 2.12 can retain both
signed zeros in some window frames, while Scala 2.13 builds and the GPU
implementation deduplicate them.
This change excludes `-0.0` from the float special cases only within
`test_window_aggs_for_rows_collect_set` when it runs against Scala 2.12.
The shared collect-set generator is unchanged, so other tests retain
their existing coverage. Scala 2.13 continues to exercise both
signed-zero values. No runtime behavior changes.
Testing:
- `python3 -m py_compile
integration_tests/src/main/python/window_function_test.py`
- Spark 3.5.9 / Scala 2.12 CPU A/B validation with
`DATAGEN_SEED=1740159968`: the original generator produced 71
signed-zero duplicate window frames out of 2,048 rows; the PR-localized
generator produced 0
- Spark 3.5.6 / Scala 2.12 GPU integration test with
`DATAGEN_SEED=1740159968` and `TZ=America/Punta_Arenas`: 1 passed
- Spark 4.0.1 / Scala 2.13 package build: success
- Spark 4.0.1 / Scala 2.13 GPU integration test with the same seed and
time zone: 1 passed
- Other window collect-set candidates using the shared generator on
Spark 3.5.6 / Scala 2.12: 3 passed
- Representative hash groupby/reduction collect-set suites on Spark
3.5.6 / Scala 2.12: 42 passed
### Checklists
Documentation
- [ ] Updated for new or modified user-facing features or behaviors
- [x] No user-facing change
Testing
- [x] Added or modified tests to cover new code paths
- [ ] Covered by existing tests
(Please provide the names of the existing tests in the PR description.)
- [ ] Not required
Performance
- [ ] Tests ran and results are added in the PR description
- [ ] Issue filed with a link in the PR description
- [x] Not required
---------
Signed-off-by: Chong Gao <chongg@nvidia.com>1 parent 4383de9 commit 8e360c1
1 file changed
Lines changed: 19 additions & 2 deletions
Lines changed: 19 additions & 2 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
23 | 23 | | |
24 | 24 | | |
25 | 25 | | |
26 | | - | |
| 26 | + | |
27 | 27 | | |
28 | 28 | | |
29 | 29 | | |
| |||
2377 | 2377 | | |
2378 | 2378 | | |
2379 | 2379 | | |
| 2380 | + | |
| 2381 | + | |
| 2382 | + | |
| 2383 | + | |
| 2384 | + | |
| 2385 | + | |
| 2386 | + | |
| 2387 | + | |
| 2388 | + | |
| 2389 | + | |
| 2390 | + | |
| 2391 | + | |
| 2392 | + | |
| 2393 | + | |
| 2394 | + | |
| 2395 | + | |
| 2396 | + | |
2380 | 2397 | | |
2381 | | - | |
| 2398 | + | |
2382 | 2399 | | |
2383 | 2400 | | |
2384 | 2401 | | |
| |||
0 commit comments