Related to #1507 and pola-rs/polars#12389
Problem
Current Lance can expose nearest search indirectly through default_scan_options and pl.scan_pyarrow_dataset. However, this makes the search configuration part of the dataset object and does not provide a clear Polars-facing API or an explicit prefilter/postfilter contract. A Lance-owned scan_nearest integration would make those semantics direct and discoverable.
Why not only default_scan_options?
default_scan_options attaches search configuration to the dataset rather than to an individual Polars query. A dedicated integration would make each nearest-search request query-scoped and would define whether filters are executed by Lance before retrieval or by Polars afterward.
Proposed capability
Add optional Polars integration in Lance. For instance:
from lance.polars import scan_nearest
lf = scan_nearest(
source,
vector_column="embedding",
query=query,
k=10,
metric="cosine",
prefilter="category = 'docs'",
use_index=True,
)Return polars.LazyFrame through current Polars IO-plugin API. Import Polars only when integration is used.
Prototype result
Research prototype using Polars 1.43.0 and Lance 0.38.2 passed:
- exact output parity with direct Lance scanner
- distinct prefilter and postfilter behavior
- correct filter/limit ordering through register_io_source
- projection pushdown
- invalid-prefilter failure
- indexed plan containing ANNSubIndex, ANNIvfPartition, and ScalarIndexQuery
No Polars core change was required for initial scope.
Related to #1507 and pola-rs/polars#12389
Problem
Current Lance can expose nearest search indirectly through
default_scan_optionsandpl.scan_pyarrow_dataset. However, this makes the search configuration part of the dataset object and does not provide a clear Polars-facing API or an explicit prefilter/postfilter contract. A Lance-ownedscan_nearestintegration would make those semantics direct and discoverable.Why not only default_scan_options?
default_scan_optionsattaches search configuration to the dataset rather than to an individual Polars query. A dedicated integration would make each nearest-search request query-scoped and would define whether filters are executed by Lance before retrieval or by Polars afterward.Proposed capability
Add optional Polars integration in Lance. For instance:
Return polars.LazyFrame through current Polars IO-plugin API. Import Polars only when integration is used.
Prototype result
Research prototype using Polars 1.43.0 and Lance 0.38.2 passed:
No Polars core change was required for initial scope.