Please describe why this is necessary.
delta-spark can delegate scan planning to a catalog or planning service, which returns the files a query may read (delta-io/delta#5623).
That issue's was targeted at Spark and the kernel has no equivalent, An engine that already knows which files a scan needs cannot tell the Kernel
Two existing area I've found in my search:
OpaquePredicateOp (#85, via #686, #980, #2627) lets an engine plug in its own "could this file match?" check but it only receives file stats and partition values (no path or deletion vector), so it can prune by value but cannot name files.
SnapshotBuilder::with_log_tail feeds commits into log replay. It is input to replay, not a filter on its output, and it takes commit files, not data files.
Describe the functionality you are proposing.
Standard method for the Kernel to receive a list of files to scan (for security or performance),
the Kernel then applies on top of its normal log replay.
-
A trait, ScanPlanner, with one method: given the table root, the snapshot version and
the scan's predicate, return the set of files the scan may read.
-
A builder option, ScanBuilder::with_planner(...). Without it, nothing changes.
-
Kernel replays the log exactly as it does today, then keeps only the files that are both
in the log and in the planner's list. The planner can remove files; it cannot add any.
Deletion vectors, partition values, stats and transforms still come from the log.
-
Files are matched on Kernel's existing identity, path plus deletion vector id, so a plan
that names a path with the wrong or missing deletion vector does not match.
-
The predicate is passed as Kernel's own Predicate type. How an implementation
serializes or evaluates it is up to the implementation.
Target is to return Files as a unit
Additional context
Prototype PR #3295
Please describe why this is necessary.
delta-spark can delegate scan planning to a catalog or planning service, which returns the files a query may read (delta-io/delta#5623).
That issue's was targeted at Spark and the kernel has no equivalent, An engine that already knows which files a scan needs cannot tell the Kernel
Two existing area I've found in my search:
OpaquePredicateOp (#85, via #686, #980, #2627) lets an engine plug in its own "could this file match?" check but it only receives file stats and partition values (no path or deletion vector), so it can prune by value but cannot name files.
SnapshotBuilder::with_log_tail feeds commits into log replay. It is input to replay, not a filter on its output, and it takes commit files, not data files.
Describe the functionality you are proposing.
Standard method for the Kernel to receive a list of files to scan (for security or performance),
the Kernel then applies on top of its normal log replay.
A trait,
ScanPlanner, with one method: given the table root, the snapshot version andthe scan's predicate, return the set of files the scan may read.
A builder option,
ScanBuilder::with_planner(...). Without it, nothing changes.Kernel replays the log exactly as it does today, then keeps only the files that are both
in the log and in the planner's list. The planner can remove files; it cannot add any.
Deletion vectors, partition values, stats and transforms still come from the log.
Files are matched on Kernel's existing identity, path plus deletion vector id, so a plan
that names a path with the wrong or missing deletion vector does not match.
The predicate is passed as Kernel's own
Predicatetype. How an implementationserializes or evaluates it is up to the implementation.
Target is to return Files as a unit
Additional context
Prototype PR #3295