Skip to content

feat: add ScanBuilder::with_planner to restrict a scan's file set - #3295

Open
JohnEarle wants to merge 1 commit into
delta-io:mainfrom
JohnEarle:scan-planner
Open

feat: add ScanBuilder::with_planner to restrict a scan's file set#3295
JohnEarle wants to merge 1 commit into
delta-io:mainfrom
JohnEarle:scan-planner

Conversation

@JohnEarle

Copy link
Copy Markdown

What changes are proposed in this pull request?

Prototype for #3294 (for design review and discussion).

Adds an optional hook an engine can use to restrict the set of data files a scan reads,
the Kernel side of what delta-spark's ServerSidePlanningClient does (delta-io/delta#5623).

  • ScanPlanner::plan(table_root, version, predicate) -> HashSet<FileActionKey>
  • ScanBuilder::with_planner(...); without a planner nothing changes
  • One check in AddRemoveDedupVisitor::is_valid_add: an Add the planner did not name is
    dropped, after the key is recorded for dedup

The planner's answer is compared against the log replay, so it can remove files but never add
one; DVs, partition values, stats and row filtering are unchanged.

Files are identified by the existing FileActionKey ((path, deletionVector.uniqueId)).

The planner gets the predicate as the caller set it; how it serializes or evaluates it is its own concern.

A planner error fails the scan.

parallel_scan_metadata and the declarative plan reject a planner rather than ignore it.

This PR affects the following public APIs

New: scan::ScanPlanner, scan::ScanPlannerRef, scan::FileActionKey (re-export),
ScanBuilder::with_planner.

How was this change tested?

13 new unit test cases:

  • absent planner unchanged
  • subset
  • unknown path not added
  • DV identity (wrong or missing DV id does not match)
  • error propagation
  • planner inputs including under column mapping
  • statically false predicate
  • parallel path rejection
  • Full suite
  • clippy -D warnings
  • rustdoc pass

@nicklan

nicklan commented Sep 10, 2026

Copy link
Copy Markdown
Member

@JohnEarle thanks for the PR! Are you ready to have someone review this? If not could you move it to draft, otherwise we can find folks to take a look.

@nicklan
nicklan requested a review from jtuglu1 September 10, 2026 18:51
@JohnEarle

Copy link
Copy Markdown
Author

@JohnEarle thanks for the PR! Are you ready to have someone review this? If not could you move it to draft, otherwise we can find folks to take a look.

Hey @nicklan, I'm happy to proceed with a review,
I think this is a lean starting implementation, any design feedback would be appreciated.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants