Commit 41b2739
[SPARK-58064][SQL] Add filter pushdown for BIN BY
### What changes were proposed in this pull request?
This PR adds `BinBy` to `PushPredicateThroughNonJoin.canPushThrough`, so the generic unary-node pushdown arm relocates deterministic predicates below `BIN BY`.
- A predicate on a forwarded pass-through or range column pushes below the operator: `BIN BY` replicates those columns unchanged across every sub-row of an input row, so filtering before binning is equivalent to filtering after, and pushing avoids expanding rows that would be discarded.
- A predicate on a scaled DISTRIBUTE or appended column cannot push, and stays above: those are produced attributes with fresh `ExprId`s not in the child output, so `pushDownPredicate` never treats them as a subset of the child.
This is safe because of the produced-attributes shape from SPARK-57858. Before that change the output DISTRIBUTE column carried the child's `ExprId`, so a predicate on it would have pushed below and filtered the unscaled value.
### Why are the changes needed?
`BIN BY` is row-multiplying, so pushing a predicate on a pass-through / range column below the operator filters input rows before binning and avoids wasted expansion (e.g. `WHERE host = '...'` or a time-range predicate). `Generate` is in the same allowlist for the same reason.
### Does this PR introduce _any_ user-facing change?
No. `BIN BY` is gated off by default (`spark.sql.binByRelationOperator.enabled`, SPARK-57440). This is an optimizer-only change and does not alter query results. A predicate pushed below the operator can filter out an inverted-range row before it is processed, so whether `BIN_BY_INVALID_RANGE` is raised may depend on the plan, as with any data-dependent error on a discarded row.
### How was this patch tested?
- `FilterPushdownSuite`: a predicate on a pass-through column pushes below `BinBy`; a predicate on a produced (fresh-`ExprId`) appended column stays above.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Anthropic)
Closes #57159 from vranes/bin-by-filter-pushdown.
Authored-by: Nikolina Vraneš <nikolina.vranes@databricks.com>
Signed-off-by: Wenchen Fan <wenchen@databricks.com>1 parent 710b3c4 commit 41b2739
2 files changed
Lines changed: 47 additions & 0 deletions
File tree
- sql/catalyst/src
- main/scala/org/apache/spark/sql/catalyst/optimizer
- test/scala/org/apache/spark/sql/catalyst/optimizer
Lines changed: 1 addition & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
2311 | 2311 | | |
2312 | 2312 | | |
2313 | 2313 | | |
| 2314 | + | |
2314 | 2315 | | |
2315 | 2316 | | |
2316 | 2317 | | |
| |||
Lines changed: 46 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1644 | 1644 | | |
1645 | 1645 | | |
1646 | 1646 | | |
| 1647 | + | |
| 1648 | + | |
| 1649 | + | |
| 1650 | + | |
| 1651 | + | |
| 1652 | + | |
| 1653 | + | |
| 1654 | + | |
| 1655 | + | |
| 1656 | + | |
| 1657 | + | |
| 1658 | + | |
| 1659 | + | |
| 1660 | + | |
| 1661 | + | |
| 1662 | + | |
| 1663 | + | |
| 1664 | + | |
| 1665 | + | |
| 1666 | + | |
| 1667 | + | |
| 1668 | + | |
| 1669 | + | |
| 1670 | + | |
| 1671 | + | |
| 1672 | + | |
| 1673 | + | |
| 1674 | + | |
| 1675 | + | |
| 1676 | + | |
| 1677 | + | |
| 1678 | + | |
| 1679 | + | |
| 1680 | + | |
| 1681 | + | |
| 1682 | + | |
| 1683 | + | |
| 1684 | + | |
| 1685 | + | |
| 1686 | + | |
| 1687 | + | |
| 1688 | + | |
| 1689 | + | |
| 1690 | + | |
| 1691 | + | |
| 1692 | + | |
1647 | 1693 | | |
0 commit comments