[SPARK-59250][SQL] Translate an empty runtime IN filter to AlwaysFalse in DSv2 runtime filtering - #58524
Closed
pan3793 wants to merge 3 commits into
Closed
[SPARK-59250][SQL] Translate an empty runtime IN filter to AlwaysFalse in DSv2 runtime filtering#58524pan3793 wants to merge 3 commits into
pan3793 wants to merge 3 commits into
Conversation
…e in DSv2 runtime filtering Assisted-by: Claude Fable 5
…tering Assisted-by: Claude Fable 5
pan3793
force-pushed
the
dpp-empty-in
branch
from
September 4, 2026 20:43
d7b2aaa to
5ccb9f1
Compare
Member
Author
|
cc @szehon-ho |
szehon-ho
approved these changes
Sep 4, 2026
szehon-ho
left a comment
szehon-ho
left a comment
Member
There was a problem hiding this comment.
Optimization makes sense to me
| spark.conf.set("spark.sql.catalog.testcat", classOf[InMemoryTableWithV2FilterCatalog].getName) | ||
| } | ||
|
|
||
| import testImplicits._ |
Member
There was a problem hiding this comment.
nit: can we put it on top or more natural place ?
Member
Author
There was a problem hiding this comment.
Moved the import and helper into the shared DynamicPartitionPruningV2Suite.
szehon-ho
reviewed
Sep 4, 2026
| scans.head | ||
| } | ||
|
|
||
| test("SPARK-59250: DPP prunes all DSv2 partitions when the runtime IN filter is empty", |
Member
There was a problem hiding this comment.
Could we also cover the SupportsRuntimeFiltering (V1 Filter) path? This PR updates InMemoryBaseTable.InMemoryBatchScan to handle AlwaysFalse, but these tests run only with InMemoryTableWithV2FilterCatalog, so they exercise InMemoryV2FilterBatchScan and leave the V1 compatibility path untested. It may be possible to share the tests with DynamicPartitionPruningV2Suite so both implementations run them.
Member
Author
There was a problem hiding this comment.
Moved the tests into DynamicPartitionPruningV2Suite, so they now run through V1 Filter, V2 Predicate, and Catalyst runtime filtering.
Assisted-by: Claude Fable 5
uros-b
approved these changes
Sep 5, 2026
Member
|
Thank you @pan3793 and @szehon-ho! |
pan3793
added a commit
that referenced
this pull request
Sep 7, 2026
…e in DSv2 runtime filtering ### What changes were proposed in this pull request? In `DataSourceV2Strategy.translateRuntimeFilterV2`, match an `InSubqueryExec` whose subquery result is empty before the pushable-column check, and translate it to `AlwaysFalse`. `expr IN ()` is false for every row regardless of `expr`, so the translation is valid even when `expr` itself is untranslatable, e.g. a join key wrapped in a cast inserted by type coercion (`cast(col as bigint) IN dynamicpruning#x`). The test connector scans `InMemoryTableWithV2Filter` and `InMemoryBaseTable.InMemoryBatchScan` are updated to honor `AlwaysFalse` by pruning all partitions. ### Why are the changes needed? When the build side of a DPP join produces no rows, the runtime filter degenerates to `key IN ()`. Today this is either emitted as a zero-value `IN` predicate that connectors cannot interpret meaningfully (Spark's own `InMemoryTableWithV2Filter` ignores it: its `filter()` requires `children().length > 1`), or, when the key is wrapped in a cast, dropped entirely with "Can't translate ... unsupported expression". Either way the scan reads every partition to produce zero output rows. Translating to `AlwaysFalse` prunes everything and makes the connector contract explicit: connectors receive a well-defined predicate instead of a degenerate IN. ### Does this PR introduce _any_ user-facing change? No correctness change. DSv2 scans receiving an empty runtime IN filter can now prune all partitions instead of reading everything. ### How was this patch tested? New tests in `DynamicPartitionPruningV2Suite` covering a bare pruning key and a cast-wrapped pruning key; both assert `filteredPartitions` is empty. The tests run through V1 `Filter`, V2 `Predicate`, and Catalyst runtime filtering. Previously all 25 partitions were read. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Fable 5 Closes #58524 from pan3793/dpp-empty-in. Authored-by: Cheng Pan <pan3793@gmail.com> Signed-off-by: Cheng Pan <chengpan@apache.org> (cherry picked from commit f3e6e17) Signed-off-by: Cheng Pan <chengpan@apache.org>
Member
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
In
DataSourceV2Strategy.translateRuntimeFilterV2, match anInSubqueryExecwhose subquery result is empty before the pushable-column check, and translate it toAlwaysFalse.expr IN ()is false for every row regardless ofexpr, so the translation is valid even whenexpritself is untranslatable, e.g. a join key wrapped in a cast inserted by type coercion (cast(col as bigint) IN dynamicpruning#x).The test connector scans
InMemoryTableWithV2FilterandInMemoryBaseTable.InMemoryBatchScanare updated to honorAlwaysFalseby pruning all partitions.Why are the changes needed?
When the build side of a DPP join produces no rows, the runtime filter degenerates to
key IN (). Today this is either emitted as a zero-valueINpredicate that connectors cannot interpret meaningfully (Spark's ownInMemoryTableWithV2Filterignores it: itsfilter()requireschildren().length > 1), or, when the key is wrapped in a cast, dropped entirely with "Can't translate ... unsupported expression". Either way the scan reads every partition to produce zero output rows.Translating to
AlwaysFalseprunes everything and makes the connector contract explicit: connectors receive a well-defined predicate instead of a degenerate IN.Does this PR introduce any user-facing change?
No correctness change. DSv2 scans receiving an empty runtime IN filter can now prune all partitions instead of reading everything.
How was this patch tested?
New tests in
DynamicPartitionPruningV2Suitecovering a bare pruning key and a cast-wrapped pruning key; both assertfilteredPartitionsis empty. The tests run through V1Filter, V2Predicate, and Catalyst runtime filtering. Previously all 25 partitions were read.Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Fable 5