Commit 4a70dc0
committed
[SPARK-58324][SQL] Drop unused sameOrderExpressions from GroupPartitionsExec k-way merge ordering
### What changes were proposed in this pull request?
`GroupPartitionsExec` builds a `SortedMergeCoalescedRDD` for the k-way merge and hands it a `LazyCodeGenOrdering` built from `child.outputOrdering`. The generated comparator (`GenerateOrdering`) only needs each `SortOrder`'s sort key (child, direction, null ordering), so this drops `sameOrderExpressions` -- planner-only metadata -- via a small `kWayMergeOrdering` helper before constructing the ordering, so it is not serialized with the RDD in every task.
### Why are the changes needed?
`sameOrderExpressions` is unused by the merge comparator and is unnecessary payload serialized with every task. It was also the vector for the `StackOverflowError` fixed in SPARK-58323 (an unforced, deeply-nested `LazyList`); not carrying it here removes this operator's exposure to any such ordering entirely (defense-in-depth), independent of that fix.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
New unit test in `GroupPartitionsExecSuite` asserting `kWayMergeOrdering` keeps the sort key but drops `sameOrderExpressions`. Existing SPARK-55715 sorted-merge tests cover comparator correctness.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 4.8
Closes #57503 from peter-toth/SPARK-58324-drop-sameorderexpressions-kway-merge.
Authored-by: Peter Toth <peter.toth@gmail.com>
Signed-off-by: Peter Toth <peter.toth@gmail.com>
(cherry picked from commit cce4355)
Signed-off-by: Peter Toth <peter.toth@gmail.com>1 parent aae1628 commit 4a70dc0
2 files changed
Lines changed: 26 additions & 1 deletion
File tree
- sql/core/src
- main/scala/org/apache/spark/sql/execution/datasources/v2
- test/scala/org/apache/spark/sql/execution/datasources/v2
Lines changed: 10 additions & 1 deletion
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
233 | 233 | | |
234 | 234 | | |
235 | 235 | | |
| 236 | + | |
| 237 | + | |
| 238 | + | |
| 239 | + | |
| 240 | + | |
| 241 | + | |
| 242 | + | |
| 243 | + | |
| 244 | + | |
236 | 245 | | |
237 | 246 | | |
238 | 247 | | |
239 | 248 | | |
240 | 249 | | |
241 | | - | |
| 250 | + | |
242 | 251 | | |
243 | 252 | | |
244 | 253 | | |
| |||
Lines changed: 16 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
48 | 48 | | |
49 | 49 | | |
50 | 50 | | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
51 | 67 | | |
52 | 68 | | |
53 | 69 | | |
| |||
0 commit comments