Skip to content

Commit 41b2739

Browse files
vranescloud-fan
authored andcommitted
[SPARK-58064][SQL] Add filter pushdown for BIN BY
### What changes were proposed in this pull request? This PR adds `BinBy` to `PushPredicateThroughNonJoin.canPushThrough`, so the generic unary-node pushdown arm relocates deterministic predicates below `BIN BY`. - A predicate on a forwarded pass-through or range column pushes below the operator: `BIN BY` replicates those columns unchanged across every sub-row of an input row, so filtering before binning is equivalent to filtering after, and pushing avoids expanding rows that would be discarded. - A predicate on a scaled DISTRIBUTE or appended column cannot push, and stays above: those are produced attributes with fresh `ExprId`s not in the child output, so `pushDownPredicate` never treats them as a subset of the child. This is safe because of the produced-attributes shape from SPARK-57858. Before that change the output DISTRIBUTE column carried the child's `ExprId`, so a predicate on it would have pushed below and filtered the unscaled value. ### Why are the changes needed? `BIN BY` is row-multiplying, so pushing a predicate on a pass-through / range column below the operator filters input rows before binning and avoids wasted expansion (e.g. `WHERE host = '...'` or a time-range predicate). `Generate` is in the same allowlist for the same reason. ### Does this PR introduce _any_ user-facing change? No. `BIN BY` is gated off by default (`spark.sql.binByRelationOperator.enabled`, SPARK-57440). This is an optimizer-only change and does not alter query results. A predicate pushed below the operator can filter out an inverted-range row before it is processed, so whether `BIN_BY_INVALID_RANGE` is raised may depend on the plan, as with any data-dependent error on a discarded row. ### How was this patch tested? - `FilterPushdownSuite`: a predicate on a pass-through column pushes below `BinBy`; a predicate on a produced (fresh-`ExprId`) appended column stays above. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Anthropic) Closes #57159 from vranes/bin-by-filter-pushdown. Authored-by: Nikolina Vraneš <nikolina.vranes@databricks.com> Signed-off-by: Wenchen Fan <wenchen@databricks.com>
1 parent 710b3c4 commit 41b2739

2 files changed

Lines changed: 47 additions & 0 deletions

File tree

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2311,6 +2311,7 @@ object PushPredicateThroughNonJoin extends Rule[LogicalPlan] with PredicateHelpe
23112311
case _: BatchEvalPython => true
23122312
case _: ArrowEvalPython => true
23132313
case _: Expand => true
2314+
case _: BinBy => true
23142315
case _ => false
23152316
}
23162317

sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/FilterPushdownSuite.scala

Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1644,4 +1644,50 @@ class FilterPushdownSuite extends PlanTest {
16441644
.analyze
16451645
comparePlans(optimizedQueryWithoutStep, correctAnswer)
16461646
}
1647+
1648+
test("push down deterministic predicate through BinBy") {
1649+
// Relation: ts_start, ts_end, value (DISTRIBUTE), label (pass-through).
1650+
val tsStart = AttributeReference("ts_start", TimestampType, nullable = false)()
1651+
val tsEnd = AttributeReference("ts_end", TimestampType, nullable = false)()
1652+
val value = AttributeReference("value", DoubleType, nullable = false)()
1653+
val label = AttributeReference("label", StringType, nullable = true)()
1654+
val relation = LocalRelation(tsStart, tsEnd, value, label)
1655+
1656+
// Produced attributes: scaled DISTRIBUTE output and appended BIN BY columns.
1657+
val scaledValue = AttributeReference("value", DoubleType, nullable = true)()
1658+
val binStart = AttributeReference("bin_start", TimestampType, nullable = true)()
1659+
val binEnd = AttributeReference("bin_end", TimestampType, nullable = true)()
1660+
val binRatio = AttributeReference("bin_distribute_ratio", DoubleType, nullable = true)()
1661+
1662+
def binByOver(child: LogicalPlan): BinBy = BinBy(
1663+
binWidthMicros = 300000000L,
1664+
rangeStart = tsStart,
1665+
rangeEnd = tsEnd,
1666+
originMicros = 0L,
1667+
distributeColumns = Seq(value),
1668+
scaledDistributeColumns = Seq(scaledValue),
1669+
appendedAttributes = Seq(binStart, binEnd, binRatio),
1670+
child = child,
1671+
timeZoneId = Some("UTC"))
1672+
1673+
// (a) A predicate on a forwarded pass-through column (label) must push BELOW BinBy.
1674+
val queryA = Filter(EqualTo(label, Literal("x")), binByOver(relation))
1675+
val optimizedA = Optimize.execute(queryA)
1676+
val expectedA = binByOver(Filter(EqualTo(label, Literal("x")), relation))
1677+
comparePlans(optimizedA, expectedA, checkAnalysis = false)
1678+
1679+
// (b) A predicate on a range column (ts_start) must push BELOW BinBy. The range columns are
1680+
// consumed by the binning logic yet forwarded unchanged in output, so their value is identical
1681+
// on every sub-row and filtering before vs. after binning is equivalent.
1682+
val tsLiteral = Literal(0L, TimestampType)
1683+
val queryB = Filter(GreaterThan(tsStart, tsLiteral), binByOver(relation))
1684+
val optimizedB = Optimize.execute(queryB)
1685+
val expectedB = binByOver(Filter(GreaterThan(tsStart, tsLiteral), relation))
1686+
comparePlans(optimizedB, expectedB, checkAnalysis = false)
1687+
1688+
// (c) A predicate on a produced (fresh-ExprId) appended column must stay ABOVE BinBy.
1689+
val queryC = Filter(GreaterThan(binRatio, Literal(0.5)), binByOver(relation))
1690+
val optimizedC = Optimize.execute(queryC)
1691+
comparePlans(optimizedC, queryC, checkAnalysis = false)
1692+
}
16471693
}

0 commit comments

Comments
 (0)